Modern search engines like Google penalize for duplicate content. They do this because they want to give credit to the original author and keep spammers or copiers from getting page rank. They also do this to separate actual content and navigation/sidebar/ and other text that is not ranked as higher value. This all helps Google provide better search results.

So whats the big deal?

 
Here is a direct quote from Google’s FAQ: How can I create a Google-friendly site?

Don’t create multiple copies of a page under different URLs. Many sites offer text-only or printer-friendly versions of pages that contain the same content as the corresponding graphic-rich pages. To ensure that your preferred page is included in our search results, you’ll need to block duplicates from our spiders using a robots.txt file.

The problem is that your Wordpress blog by default will not rank well in Google’s search engines because of the duplication problem. Fortunately Google and other search engines have provided us with a tool to inform their search engine spiders to ignore specific content.

Let me give you a few examples of where Wordpress contains the same content throughout multiple URLs.

1. www.yourblog.com has by default that last 10 or so posts. The “original content” maybe in this URL: http://www.yourblog.com/2008/04/title-of-story , but is also on your home page.

2. http://www.yourblog.com/2008/04/ - Your archive pages contain the same content as your main page with a date range.

3. http://www.yourblog.com/category/category-name/ - The same thing goes for your category pages. Every post in the category-name category will be duplicated within this URL.

4. http://www.yourblog.com/feed - All articles in their entirety are duplicated in all of the Wordpress default feeds.

5. http://www.yourblog.com/search - Of course all search results will also be duplicated content.

As you can see these URLs are all different but contain the exact same content. If you don’t want to be penalized by google you need to create a file at the root directory of your blog called, ‘robots.txt‘. This is the file that search engine spiders will be looking for and this is where you specify the rules you want them to follow.

Robots.txt

 
The following robots.txt file will pretty much restrict search engine spiders from most of the duplication problems I can think of.

User-agent: *
Disallow: /wp-
Disallow: /search
Disallow: /feed
Disallow: /comments/feed
Disallow: /feed/$
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$
Disallow: /*/*/feed/$
Disallow: /*/*/feed/rss/$
Disallow: /*/*/trackback/$
Disallow: /*/*/*/feed/$
Disallow: /*/*/*/feed/rss/$
Disallow: /*/*/*/trackback/$

Since we want the bots to follow links in our category, archive pages and certainly the home page we will have to treat them differently. We want search engine spiders to follow the links on these pages yet we don’t want the actual content indexed due to the duplication penalties.

This code will tell the bots to follow links and ignore content. Place this html code in your archive and category pages.

<meta name="robots" content="noindex,follow" />

Another line of defense for fighting against duplication on these pages is to use the Wordpress ‘more‘ function. When you add the more function in your posts Wordpress will cut the article off at that point and offer the reader a “read more” link. Not only does this help with duplication issues it will make navigating through articles much easier for your users.

The more function:

<!-- more -->

Of course Google has many ways to determine page rank and how pages are indexed. We can never be sure how Google weighs them and what other factors determine a page’s quality content. So you may find a piece of content that is duplicated all over the internet but the original is still ranked high because of link popularity or other factors.

Your entire website may also be a duplicate. Check out, Duplicate Content www vs. non-www Canonical Problems.

The strength in this lies in the fact that someone listening to authentication process knowing every detail about the process would not be able to utilize a replay attack. That is, they would not be able to authenticate if they typed in your password and one time key. This is because the key is exactly how it is written, one time use.

If someone were to find your paper with one time keys they would still not be able to log in because they do not have knowledge of your password.

Although Steve Gibson from Security Now has a more secure implementation and full documentation on how to implement it, Verisign/Paypal have a similar proprietary system in place. One of the major differences in their system is they don’t use a piece of paper but rather a digital device with keys that change based on time. They are synced with the server and display a different key based on the current date and time.

With security these days and the number of vulnerabilities I was excited to find out that Paypal offers the device for only $5.00. These devices purchased directly from Verisign are $30.00 for the football shaped one and $48.00 for the credit card style. The Paypal device is the exact same for a fraction of the price and worth every penny. The best part of the whole thing is that the device can be used for all online services that use OpenID Authentication.

Let me reiterate how cool this thing is. Someone can have full knowledge of your password but still have no access without having this physical device in their hands. On the flip side if you lose the device it is meaningless without knowledge of the account’s username and password.

They only thing draw back is having to carry around the key chain size device around. They do have a credit card style device but it is $48.00. This would be much more convenient though because you can carry it in our wallet and most people carry their wallets around.

Create your own

 
If you are interested in creating your own one-time multi-factor authentication system using paper instead of an electronic device you can check out, perfect paper passwords. Open source applications have already been implemented. Someone has already written a PAM plugin for Linux so you can authenticate SSH sessions with this method. There is even a PHP function for creating secure web logins. The method described is even more secure than Verisign but you do have to print out paper cards every 100 or so logins.

Feds: We will search through your laptop files at the border - Well this seems crazy to me. If you are traveling abroad you may be subject to search and seizure of your laptop/cellphone data. I’ll quote good ol’ Ben Franklin, “They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety.” I recommend using a program like truecrypt for whole drive encryption. I think it would also be fun to write over the “blank spots” of your hard drive with Rick Astley videos.

Microsoft device helps police pluck evidence from cyberscene of crime - Microsoft apprently made a USB device that allows law enforcement to quickly decrypt passwords and perform other forensic tasks. I am guessing it will decrypt Windows passwords easily since this has already been done, but what good is it for people that operating systems with real encryption and authentication methods?

Opera 9.5 beta 2 improves speed, adds features - The author of this artcile does his own tests and finds Opera is much faster than Firefox and the new beta version of Firefox. I have always liked Opera for many reasons and always knew opera was lighter and faster than other browsers. The artcile is a well written review of the newest version.

Error Message: Your Password Must Be at Least 18770 Characters and Cannot Repeat Any of Your Previous 30689 Passwords - An actual error in Microsoft Windows with resolution. Unbelievable!

Hackers Focus Efforts on Firefox, Safari - An intersting article about hackers targeting the ever popular Firefox and Safari internet browsers. I suspect hackers will be targeting Firefox even more in the future.

10 Tips for After You Install or Upgrade Ubuntu - If you are new to Linux you are probably using Ubuntu. If you are new to Ubuntu you should probably check this article out.

Use the YouTube API with PHP - A neat in depth article about using Youtube’s API with PHP. I havn’t thought of a way to use this yet, but very cool.

I was surprised to find that on my particular blog the smaller Adsense ad was significantly more effective than the larger 300×250 ad. This is great news for you since there will be less ad space and more content.

Here are my results for the split testing of the two ad mediums.

250×250

 
48.27% of the clicks
.59 CTR
eCPM was 171% higher than 300×250 resulting in 73.03% of the revenue.

300×250

 
51.72% of the clicks
.64 CTR
eCPM resulted in 26.97% of the revenue.

Although the 300×250 ad had a slightly higher click through rate, the 250×250 ad out performed the other ad in terms of net revenues.

Have you split tested your ads? If so please share your results.

To create an alias you need to modify ‘/home/user/.bashrc‘. Towards the bottom of the file you can add lines for aliases. Most people will find that they already have at least one default alias. Usually the ‘ls‘ command is mapped to ‘ls -G‘. The ‘-G’ flag tells the program to print directories in color.

Once the ‘.bashrc‘ file is modified you will have to re-login to that shell for the settings to take effect.

Example Aliases

 
Lets say you maintain multiple servers using ssh with different IP addresses and varying user names. You could use aliases to minimize typing and having to memorize IP addresses and names.

alias server1='ssh 192.168.1.102 -l pete'
alias server2='ssh 192.168.1.103 -l larry'
alias server3='ssh www.myremoteserver.com -l bob'

You could make a system backup script and whenever you feel like you need to make a quick unscheduled backup you could use:

alias backupNow='sh /path/to/my/backupscript.sh'

You could make an alias to show open ports in your system.

alias openports='netstat -nape --inet'

It is often helpful to make an alias for really long directories that you navigate to a lot.

alias g2path='cd /go/to/this/path/that/is/really/long/and/annoying/to/type; ls'

For the ultra paranoid you can set up an alias to wipe your hard drive.

alias wipenow='dd if=/dev/urandom of=/dev/hda'

Using aliases with variables

You can use variables in aliases if you want to get more use out of them.

alias psx="ps -auxw ¦ grep $1"

This will except a variable in the alias. For example you could type this command:

$ psx ssh

This will print out detailed information about the ssh process.

By setting variables into your alias you can make your alias have multiple uses. For example our ‘psx‘ alias can be used to find all the processes started by user, ‘mark‘.

$psx mark

If you have a long command that you type frequently consider putting it in as an alias. If you think of any useful alias ideas let us know in the comments below!

Linux

 
Start by installing gnupg
For Debian and Ubuntu systems use:

# apt-get install gnupg

Installing GnuPG on FreeBSD

 
To install GnuPG with FreeBSD’s ports package management system use:

# cd /usr/ports/security/gnupg
# make install clean

You also need to install pinentry. This is the program gnupg uses to create password phrases.

# cd /usr/local/ports/security/pinentry
# make install clean

If you don’t install this you will get an error when you run gnupg like this:

gpg-agent[13068]: can’t connect server: `ERR 67109133 can’t exec `/usr/local/bin/pinentry’: No such file or directory’
gpg-agent[13068]: can’t connect to the PIN entry module: IPC connect call failed
gpg-agent[13068]: command get_passphrase failed: No pinentry
gpg: problem with the agent: No pinentry

I am not sure why FreeBSD doesn’t mark this as a dependency.

Encrypting the File

 
To encrypt the file all you have to do is:

gpg -c someFile.txt

To decrypt you can use:

gpg someFile.gpg

You will probably want to destroy the original unencrypted file. You can Securely Wipe a File with DD. By wiping the file with DD the original file will not be easily retrievable on your hard drive. This step is usually necessary because simply deleting doesn’t actually destroy the file.

To print out the exact line with cat you can use:

$ cat -n someFile.txt | grep -w 500

This would print out the 500th line.

Command Breakdown

 
cat - this command will concatenate (print) text.
-n - the ‘n’ switch tells cat to add line numbers.
| - the | or AKA pipe will take the results from ‘cat -n’ and pipe them to grep
grep - this is a text search program. This will look for certain strings
-w - the w command tells grep to only match whole words
500 - this is the “word” we are looking for.

Adding Line Numbers to Vim

 
Open the file with Vim:

$ vim someFile.txt

Then add the show line numbers option by typing the following:

:set number

The 7 Habits of Highly Effective Linux Users - An interesting article with 7 great tips for beginners learning Linux. Although the advice the author gives is very basic and common knowledge, he explains his advice very well. I think Linux beginners would get a lot of value out of reading this.

TaoSecurity - This is a cool blog I found that discusses various security topics.

Lighttpd - Although I have yet to try this web server it looks very promising. It is supposed to be a lot lighter weight and faster than apache. I might use this on my test server.

David Walsh Blog - If you are into web development with PHP and starting to implement AJAX this site is great. It is updated daily with lots of neat tips and how tos.

Sandboxie - If you haven’t tried this program yet you should. It is a program that runs in windows that allows you to open programs in a “sandbox” or a virtual environment that does not have access to your hard drive. In other words you could open a virus with this program. After you close the sandbox the virus is like it was never there. This is useful if you want to try demo programs without the risk.

chkrootkit - This is program is pretty much essential if you are looking for root kits.

Network Calculators - If you are setting up a new router or firewall rules and you forget the binary structure of an IP address this site will help you calculate subnet masks and how many nodes are available in the network.

One way to find ports is by using the echo command and some wild cards. This is my favorite method. For example if we were looking to install the Apache web server we could use this command:

# echo /usr/ports/*/*apache*

If you are feeling lucky you can even use the change directory command to go right into the port’s directory.

# cd /usr/ports/*/*portupgrade*

An alternative way to find ports would be to use ports built in search feature.

# cd /usr/ports
# make search name=apache

You could also visit FreeBSD’s online ports search.

To stay updated with new ports you can check out FreshPorts. FreshPorts offers a service that will email you anytime there is a new version or change to a specific port. This is really helpful if you are running a production server that has a mission critical port on it, like an Apache web server.

Using one of these methods you are sure to find what you are looking for.

Fortunately FreeBSD has a great utility to keep everything updated. It’s called portupgrade. If you don’t already have it installed you can install it through ports.

# /usr/ports/ports-mgmt/portupgrade

Now before you start the portupgrade process lets find out what ports actually need updating. This way if we find any programs that we would rather have the old version we exclude it. Another reason we want to know which programs need updating is to find applications that may require special updating instructions.

To find the programs that are out of date run:

# pkg_version -v

This is also a good time to watch the list and find packages that you may no longer need.

To update all ports use:

# portupgrade -a

If you want portupgrade to ask you each time before updating you can use interactive mode.

# portupgrade -ai

Maybe you ran a portaudit and only one of the programs needs a security upgrade and you just want to upgrade that one for now.

# portupgrade -R firefox

That is about all there is to updating your entire FreeBSD system. I sure wish I could update every program in Windows with one simple command.