Python - The One True Language

I find it interesting when people bad-mouth Python - To the point where I felt the need to write this little blog article.

First, I need to state the obvious - Python is the 'One True Language'.  :-)

I say that as someone who boot-strapped himself from Commodore BASIC (PET, not that fancy C64 thing), to MS-Basic, to Turbo C. Many an hour was logged in front of this little window..

Sexy IDE for 1987

C was clearly my first true love (well perhaps Angeline who sat in front of me in Gr. 7, but she never responded to me like an 16550 UART call in C did). Compared to the other languages at the time, it gave you incredible power, without the difficulty of ASM.

I've had the (mis?)fortune to have programmed in just about every popular language at one time of another in my 30+ year career, and I must say Python is the first language that usurped C's place in my heart.

My only regret is that I didn't try Python earlier.

A few years ago I had a choice to make between Perl and Python as my next scripting language for Unix tasks that were too complex for shells or PHP, and I made the choice to go for Perl, as evidenced by the 2006 receipt still in my O'Riley Programming Perl book on my shelf.

At the time it was a close call; Perl had more popularity, and looked to be continuing to grow. This was an important point when you want to invest time in a new language (anyone remember the D Language?).  I also think I liked the camel on the Perl book cover more than the python as well... who was into snakes? (I did not realize the Monty Python reference at the time). These things do matter when you're down to brass tacks.

So the books were purchased and well read, and a few years were spent in Perl. 

I enjoyed the C-like syntax, and relaxed feel. However, it wasn't everything that I wanted.. and I'm not sure exactly how I stumbled on Python again, nor what the light bulb moment was - I worked a lot back then, so memories are fuzzy. Something was lacking, something kept me driving forward into new languages.

Whatever made me stop and reconsider, I must say I'm very happy to be in Python now - and nearly 100% of what I write today is Python.

I need a bit of Javascript for DOM work, and a bit of C for some very heavy transactional stuff, but even some of my stress tests and message passing routines are now Python.

Why do I bring this up?

I feel a number of programmers looking at Perl/Puby/Python may fall into the same situation, even today when the Python community continues to gather strength.  No one is really out there bashing Perl or Ruby, but Python has it's haters.

Some people love to badmouth Python.. I don't really understand them - I think it must be a machismo thing. The fact that it's an interpreter, and that it doesn't thread well (you shouldn't use threads boys.. that' 1990's tech) seem to enrage some.

If you're considering investing time into Python or XYZ, then my suggestion is to take the time to try Python first. I'm not going to get into a pointed discussion on the pros and cons of the various languages - That's covered in detail across the web.

Instead I'll state: They are all good languages, but if you take the time to learn Python and really 'drink the kool-aid', I think you will be rewarded.

Python brings a joy back into programming that I haven't felt since I used to sit in front of that little Turbo C IDE back in 1988. I've hit a productivity level that is a career milestone thanks to Python.

Next: PyCharm - The Turbo C IDE for Python. :-)

PS - I predict an OS entirely in Python soon - Obviously with a special interpreter, but it will be here. 

FreeBSD 9.1 to 9.2 Upgrade with freebsd-update

FreeBSD 9.2 brings a very nice update to ZFS : LZ4 compression.  Upgrading from 9.1 to 9.2 is quite easy if you have a standard install (no customization of the code).


The basic process, which should work for other versions as well is this:

1) Get your system up-to-date with the latest patches for your version of FreeBSD. This isn't necessary, but I think may avoid some problems.

freebsd-update fetch
freebsd-update install
reboot

2) Upgrade the kernel

freebsd-update upgrade -r 9.2-RELEASE
freebsd-update install
reboot

3) Update userland (finishes off the 2 part install)

freebsd-update install
reboot

4) Update pkg if needed

pkg update

5) Enjoy

With my 9.1 -> 9.2 update, I didn't have any messy mergemaster conf files to pour through, nor did my packages notify me that I needed to upgrade them - One of the more painless upgrades I've made.

pkg - FreeBSD's new Package Manager is Finally Working

I was about to write a small entry about how to hook up FreeBSD's next-generation package manager (pkg) to PC-BSD's repository, as FreeBSD wasn't supplying package updates due to the security incident that occurred in Nov 2012.

However, pkg is now able to pull properly from FreeBSD sources, and it's a nice little program.

For all those still using the old pkg_* type tools, just type pkg and follow the prompts.

So far I am finding it easier to upgrade and maintain my packages with pkg, which is always a good thing.

Boot from ZFS root with FreeBSD 9.2

(Update: The newly released FreeBSD 10 makes a ZFS setup in the menu system easy.. give it a try)

Booting from a ZFS root has always been a dream of mine, but I've never managed to make it work.

Since I'm deploying new FreeBSD 9.2 images for testing in my lab, I thought this would be a good time to give it a shot again.

I based this off the FreeBSD install page, with some simplification and a few small additions.

I took a freshly installed FreeBSD 9.2 system to act as my workstation for this, and plugged in a 8 gig USB Flash drive to be my ZFS root.

In my setup, da0 is the booted system, da1 is the blank USB Flash Drive.

I often do not use swap partitions on USB Flash, as it can prematurely wear the drive.  I make my USB Flash drives as disposable as possible, so keeping crash dumps isn't a concern.

NOTE: Solaris always called it's root ZFS pool 'rpool' and I recommend you do the same, as it's a existing standard. My use of 'zroot' here is because I was only fooling around... and seem to be too lazy to change the text below without testing for typos.

Here is what I have executed to make it work:


gpart create -s gpt da1
gpart add -b 34 -s 64k -t freebsd-boot -l boot0 da1
gpart add -t freebsd-zfs -l root0 da1
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da1

#This may fail with an "operation not permitted" error message, since the kernel likes to protect critical parts of the #disk. If this happens for you, run:
#
# sysctl kern.geom.debugflags=0x10


kldload /boot/kernel/opensolaris.ko
kldload /boot/kernel/zfs.ko

zpool create zroot da1p2
# DO NOT FORGET TO MAKE IT P2! Specify da1, and zfs will kill your gpt tabes
zpool set bootfs=zroot zroot
zfs set checksum=fletcher4 zroot


#Make sure your FreeBSD 9.2 CD is in your drive

mount -t cd9660 /dev/cd0 /mnt

cd /mnt/usr/freebsd-dist

cat base.txz | tar --unlink -xpJf - -C /zroot
cat kernel.txz | tar --unlink -xpJf - -C /zroot
cat doc.txz | tar --unlink -xpJf - -C /zroot
cat lib32.txz | tar --unlink -xpJf - -C /zroot
cat ports.txz | tar --unlink -xpJf - -C /zroot
cat src.txz | tar --unlink -xpJf - -C /zroot

chroot /zroot


echo 'zfs_enable="YES"' > /etc/rc.conf
echo 'sshd_enable="YES"' >> /etc/rc.conf
echo 'hostname="freebsd92zfs"' >> /etc/rc.conf

echo 'zfs_load="YES"' > /boot/loader.conf
echo 'vfs.root.mountfrom="zfs:zroot"' >> /boot/loader.conf

tzsetup

cd /etc/mail
make aliases

exit
cp /boot/zfs/zpool.cache /zroot/boot/zfs/zpool.cache
zpool export zroot




I then pull the USB Flash Drive, pop it into a spare machine, and presto, it's booting.


I'll fill in more about this as I play with ZFS on root for my USB Flash Drive systems.

(FIXED) Network mbuf Leak / Exhaustion in FreeBSD 9.0 / 9.1

UPDATE: Patch included below

There is a rather annoying bug floating around in FreeBSD 9.0 / 9.1 regarding network mubf leaking to the point of mbuf exhaustion.

These is a problem report (PR) filed about this from last year, but it looks to be abandoned.

http://www.freebsd.org/cgi/query-pr.cgi?pr=165903&cat=

 I am experiencing the same mbuf leak on fresh 9.1-RELEASE machines (AMD64). Most of my machines are ESXi 5.1 VM's running the e1000 (em0) NIC. This VM is stock, just one freebsd-update done, nothing custom.

 I have also experienced this condition on an older 9.0-STABLE from Jul 1st 2012. I did not notice it much before that date, but I can't tell for sure. I have a few machines on that build that I still use, so confirmation was easy.

 I do not experience the error if I load up vmware tools and use the vmx3f0 adapter, it's just with em0.

 I have set the mbufs to very high numbers via sysctl kern.ipc.nmbclusters=322144 to buy more time between lockups/crashes. Most often the systems stay functional, they just need a reboot or more mbufs if I add them. Some times the servers lock up or crash as I ifconfig down/up the adapter or attempt to add more mbufs via sysctl.

Is anyone else able to reproduce this?

I have attempted to update the PR or post to the list, but the freebsd.org server and my mail server no longer seem to get along. I'll have to troubleshoot that later this week.

UPDATE Apr 19th 2013:

Gleb Smirnoff was kind enough to quickly forward me a patch that fixed the problem for me. You will need to apply this to usr/src/sys/netinet/if_ether.c

I've now run for 2 days, and my mbufs have not increased at all.  Thanks for the quick response from the FreeBSD-Stable list.

Index: if_ether.c
===================================================================
--- if_ether.c    (revision 249327)
+++ if_ether.c    (working copy)
@@ -558,13 +558,13 @@ in_arpinput(struct mbuf *m)
     if (ah->ar_pln != sizeof(struct in_addr)) {
         log(LOG_NOTICE, "in_arp: requested protocol length != %zu\n",
             sizeof(struct in_addr));
-        return;
+        goto drop;
     }

     if (allow_multicast == 0 && ETHER_IS_MULTICAST(ar_sha(ah))) {
         log(LOG_NOTICE, "arp: %*D is multicast\n",
             ifp->if_addrlen, (u_char *)ar_sha(ah), ":");
-        return;
+        goto drop;
     }

     op = ntohs(ah->ar_op);