zero indexed

This is just a blog to try and spread some of the knowledge that has been freely given to me by the wider community, without which I'd get absolutely nothing accomplished. I hope this benefits some of you out there.

Tuesday, August 25, 2009

Command Line English -> Foreign Language Dictionary



I have been studying Japanese for years. I love trying to learn the language, but it takes constant work. Lately what I've been trying to do is listen to NHK radio(their national news network) online. This has been great but since I'm not nearly fluent I have a lot of words I want to look up. And being at work I need to be able to look up something fast and continue on with what I'm doing i.e. resubmitting a search to an online dictionary is not going to cut it.

I found a great online dictionary at freedict, but as I mentioned above, it is a bit cumbersome. So I wrote a small python script to do the look up on that site for me. Put this on your path and you've got a commandline multi-language dictionary. This particular site does a lot of different translation, so I am going to modify my script so you can specify what exactly you want to translate and expand away from just Japanese to English. But for now, enjoy:


#!/usr/local/bin/python2.5

import urllib
import sys
import re

pattern = re.compile('')

data = urllib.urlencode({"search":sys.argv[1],"exact":"true","max":"10","to":"English","from":"Japanese","fname":"eng2jap2","back":"jap.html"})

file = urllib.urlopen("http://www.freedict.com/onldict/onldict.php",data)
results = pattern.sub('**',file.read())

pattern = re.compile('<(.|\n)*?>')
results = pattern.sub('',results)

pattern = re.compile('\s*\n+')
results = pattern.sub('\n',results)

pattern = re.compile('New search')
results = pattern.sub('',results)

pattern = re.compile('Online Dictionary Search Results')
results = pattern.sub('',results)

pattern = re.compile('\s\s\sJapanese')
results = pattern.sub('',results)

pattern = re.compile('\s\s\sEnglish')
results = pattern.sub('\n',results)

pattern = re.compile(' ')
results = pattern.sub('',results)

results = results.split('\n')
iterator = iter(results)

while True:
try:
line = iterator.next()
if '**' in line:
pattern = re.compile('\s*[A-Za-z]+')
match = pattern.match(line)
if match is not None:
line = line + iterator.next()
line = line.replace('**',' ')
print line
except StopIteration:
break




It is not perfect yet, but it'll do the job 95% of the time.

Friday, August 14, 2009

CPAN, Perl's Mothership



I recently wrote post about setting up a new mail server and how during that process I needed to install cpan as well as compile and install a perl module from source.

1. Download the zip/tar archive and uncompress e.g. tar xzvf my-perl-module.tar.gz

2. cd my-perl-module. Read the README and INSTALL files.

3. Run perl Makefile.pl PREFIX=/my/perl/directory.

4. Run 'make' and then 'make test'

5. Run 'make install'

6. Enjoy your new, bitchingly cool, perl module.

If this explanation is inadequate or you want a second opinion, check out cpan for more detailed, and probably correct, instructions.

On a related note, you can install modules using cpan itself. cpan was already installed, but not configured, with CentOS 5, but any modern version of linux will either come with it or have a way to easily install with apt-get/yum, etc.

The configuration I did was just the default, but I did have to install gcc before I could finish.

Once you do finish configuring cpan, to run it is dead easy: cpan

Running 'help' will get you a list of commands for the cpan shell. I just used it for installing modules so all that was necessary is just entering 'install your_desired_module' e.g. 'install DBI'

You can find the exact name of the module you need at search cpan.

That should take care of everything for you.

CentOS 5 Mail Server: Postfix, Dovecot, SpamAssassin, MailScanner



I have recently had the pleasure of setting up a new mail server. The old machine was not doing a good enough job of cutting down on spam and it needed an update. There are some great tutorials out there, but as is often the case there are minor changes that need to be made. I thought I'd document my experience to help anybody else along.

Following the tutorial regarding the setting up of postfix and dovecot just a matter of typing. I skipped Squirelmail and setting up the firewall as the mail server was going to be behind one already. MailScanner, although this should've bee a piece of cake, was giving me some grief. Nearly all of it boiled down to permissions. I knew they were wrong, but I wasn't given any indiction as to what about them was wrong i.e. who was the service running as, what permission did they need and on what file. MailScanner appeared to be working in that it scanned for viruses and then deposited the mail for postfix to process, but that was as far as the mail was getting. After much digging and playing around with configuration files I found out that the owner on /var/spool/postfix/active/random/file was actually root. I don't know what the file is, but it is necessary to deliver your mail, and the owner must be postfix, not root. After fixing that, one of my problems went away. The other problem was with Clamav.

For some reason installing Clamav via yum on CentOS 5 gives you a /etc/clamd.conf file that is slightly different than what you'd expect. The location of the socket that clamd will use is different than in the tutorial and will have to be changed either in /etc/MailScanner/MailScanner.conf or /etc/clamd.conf so they match. The exact part you'll need to change in MailScanner.conf is the Clamd Socket. It defaults to /tmp/clamd.sock, or something like that but in the clamd.conf file it will be in /var/run/clamav/clamd.sock. Go figure.

My Recommendation on this is to follow the tutorial to the letter as far as creating directories goes. If the permissions are at all wrong it will become obvious very quickly and be somewhat difficult to track down.

One thing to note, make sure to get all the perl modules needed before installing MailScanner. If you don't you'll have to look in /var/log/maillog to see which module you're missing and install it before you can go any further. I have little experience with perl but I did learn how to setup and configure cpan as well as install a module from source.



Friday, July 31, 2009

Behold: Awk



I've never really used awk before, but today I had reason to to help with some iptable rules I was using, and for my purposes it was quite easy and straight forward (even a bit of fun)

I had written some rules to do some port forwarding on the iptables nat table, but may have found a
problem with the routing as running 'up2date' no longer works.

I found some rules that should allow connections from rhn.redhat.com, where up2date needs to connect to in order to run properly, but they had the ip address hard coded. While this probably will be ok for a while, the ip address could easily change so I wanted something more dynamic.

All of the following is in a bash script that builds my iptable rules. I've only included the relevent parts below. This is what I came up with:

...

redhatip=`host rhn.redhat.com | head -n 1 | awk '{print $4}' | awk -F "\." ' $1 <= 255 && $2 <=255 && $3 <= 255 && $4 <= 255'`

if [ -z "$redhatip" ]
then
echo "invalid IP address for rhn.redhat.com! Please double check the script for correctness."
exit
fi

iptables -A OUTPUT -o eth0 -p tcp -d $redhatip -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -A INPUT -i eth0 -p tcp -s $redhatip -m state --state ESTABLISHED -j ACCEPT

I've included the rules in the iptables just for completeness incase anybody else needs to know how to get up2date to start working again.

Please note that I haven't had a chance to test the iptable rules yet, so you are on your own as far as that is concerned, but as far as getting the ip address from using host and then checking it for correctness, that all works. Enjoy!

Thursday, July 16, 2009

Programmers Take Note



I recently read a blog entry on Reddit that got me thinking. The gist of it was something like this: take notes of everything you want to know how to do because it will become your most valuable tool.

I truly believe that. They don't have to be fancy or in order, they just have to be there and searchable by using grep. A record of your experience will become invaluable and I have started making notes myself.

Here is a small bash script I wrote to help make making notes easier:

#!/bin/bash
today="note_`eval date +%Y_%m_%d`"
filename="/home/dir/notes/$today"
if [ -z "$1" ]
then
exec vi $filename
else
echo -e "\n\n" >> $filename
`cat $1 >> $filename`
exec vi $filename
fi

This will allow you to either just type "makenote" and start/add to today's note, or "makenote your_script.sh" to append the script to the bottom of your new/current note. From there you can add your comments about the script.

Put a symbolic link to this script, or simply copy it, into your path (probably /usr/bin will do)
and you can use the command makenote from anywhere and the note will go to the right place.

Here is a small example of some of the notes I've made, maybe it'll inspire some of you:

FIND FILE BY NAME

About

The `locate` command will find all files that have the name you
specify. Probably the results will be too much, so pipe it through
as many greps as you need to find what you're looking for.

Example

locate python | grep sqlite | grep transactions


MYSQL SHOW COLUMNS

Example

SHOW COLUMNS FROM mytable FROM mydb;
SHOW COLUMNS FROM mydb.mytable;


FIND ALL PORTS CURRENTLY LISTENING

Run

nestat -an|grep -i listen

Extra

Most port numbers are listed in /etc/services


The only problem with taking notes is that these days we all work on many different machines. Some are at home, others at multiple work stations. How do we keep them all in sync so we don't lose any? Good question. Maybe something with the google docs api to sync our notes across all computers, like a simple command to grab the latest in google docs and then write them back when you're done. If anyone has an idea to solve this problem please let me know!

Friday, June 12, 2009

Bash Script for Comparing Two Conf Directories



I have had a small problem of screwing up my jboss the other day. To get it working again I had to copy my co-worker's jboss directories and start using that. Slight problem with that, all the conf files will now be specific to him and not me e.g. my jndi ports, etc. So I had to go through by hand and fix all of the conf files I could find. Luckily I had a backup of my old conf files to check against. In an effort to track down the differences I wrote the following bash script.

My bash is not good, but it seems to do the trick and will compare files in two directories who's only difference is the path.

#!/bin/bash

#List the files that end with 'xml' in the final directory of $1 (you specify $1
#as the first arg on the command line)
#Note that when you list with a wild card *nix will list out the full path. If you know how to disable that #for ls, then you can use that instead, but I didn't see anything in the man pages. Therfore, strip it #out with a quick and dirty regex

ls /home/blah/jboss/server/all/$1/*xml > dir

orig_jboss="/home/
blah/jboss-4.2.3.GA.20090611/server/all/$1/"
new_jboss="/home/
blah/jboss/server/all/$1/"

#note the >>. If you don't put both it will only write the last one into the new file.
#>> will force an append
for file in `cat dir`; do echo ${file#/home*$1/} >> actual_dir ; done

for i in `cat actual_dir`; do diff $new_jboss$i $orig_jboss$i ; done

#If you don't do this and run the script a few times, you'll keep appending more
#things into 'actual_dir'
rm dir actual_dir

If everything is the same you shouldn't see any output. Otherwise you'll get a list of all the differences without having to do much by hand.

Tuesday, June 9, 2009

Python Simple HTTP Server



Recently I have gotten into using python for as many things as I can. So far the results have been good; I like the significant white space, and most functions are fairly intuitive.

Python also makes nearly a program as short as possible. Starting in version 2.5 there is a very easy way to start a http server for testing and sharing a file with a friend. I've cobbled together a few examples that get progressively more involved. I take no credit for writing these commands/scripts.

python -m SimpleHTTPServer 9914 

OR
python -c "import SimpleHTTPServer;SimpleHTTPServer.test()"

And finally:

#!/usr/bin/python

import BaseHTTPServer, SimpleHTTPServer

import os
import sys

def run(server_class=BaseHTTPServer.HTTPServer,
handler_class=SimpleHTTPServer.SimpleHTTPRequestHandler):
print 'Server version:',handler_class.server_version

port=8000

if len(sys.argv)>1:
if sys.argv[1].isdigit():
port=int(sys.argv[1])
server_address = ('', port)

httpd = server_class(server_address, handler_class)

myurl='http://localhost:'+str(server_address[1])+'/'
print 'Your Server is running on:',myurl
print 'and serving files from:',os.getcwd(),'and below.'
print 'To stop the server, type ^C.'

if 'b' in sys.argv:
print 'Trying to start webbrowser...'
import webbrowser
webbrowser.open(myurl)

httpd.serve_forever()

run()

Followers