On September 1, 2010, I uploaded Release Candidate 9 and let it go into the wild. In addition to adding GENES 1.20 input and output, the changes include some minor documentation updates and a few small code fixes.

In Release Candidate 10, which will be released this week, all I’ve done is add an option to pyp_nrm.fast_a_matrix() to add coefficients of inbreeding read from the pedigree file to the diagonal elements for founders when forming the relationship matrix. I need to update the documentation and check the files once more because I think that there are still some stray hard-coded debugging messages which need to be removed.

The only other thing I’m thinking of adding is support for unknown parent groups, and possibly adding methods to the NewPedigree class that support adding (merging) and subtracting (splitting) pedigrees, but I think that’s going to get punted to the 2.1 development branch because it’s a much harder problem than it may sound on the surface.

I still need bug reports and feature requests. I’ll do my best to fix bugs promptly. I can’t  make any promises about feature requests. My thinking right now is that I need to focus on polishing things. The demographics (pyp_demog), graphics (pyp_graphics) and reporting (pyp_reports) modules all could use a lot of fine-tuning to make them more useful. The documentation always needs work. I’m still concerned about the implementation of the algorithms in pyp_metrics for calculating effective ancestor numbers. They’re much less fun that implementing new features, but probably necessary. And then there’s the whole issue of dense SNP data…

These errors, in a way, exposed the tip of an iceberg: there is not nearly enough unit testing in PyPedal. Many of the example programs were written a long time ago (about three years) and not checked systematically since. This means that some programs, such as examples/new_amatrix.py, were still using methods that no longer exist (I’m thinking specifically about the info() method of NewAMatrix objects). Heck, that program still imported Numeric! So it’s a great idea to have examples, but it doesn’t help users when they run the program and everything fails. It makes the whole package look shoddy and undependable, and who wants to trust their work to that kind of code? If Matthieu hadn’t e-mailed me about those problems I’d still be shipping broken code. I hate writing unit tests as much as the next guy, but PyPedal’s grown to the point that it’s too big for me to keep track of, so I need a way to automate testing. So, it’s getting to be unit test time. With that and the work I want to do on the graphics module there’s plenty of work in the queue for version 2.0.1.

If you have any problems installing PyPedal please let me know so that I can fix them before the final release.

>>> # Load PyPedal
>>> import PyPedal
>>> # Specify program options -- note that we're accessing an SQLite database
>>> options = {}
>>> options['messages'] = 'verbose'
>>> options['pedfile'] = 'hartlandclark.ped'
>>> options['pedname'] = 'Pedigree from van Noordwijck and Scharloo (1981)'
>>> options['pedformat'] = 'asdb'
>>> options['pedigree_is_renumbered'] = 1
>>> options['database_name'] = 'new_db_test'
>>> options['dbtable_name'] = 'test'
>>> options['database_type'] = 'sqlite'
>>> # Now we load the pedigree
>>> example = pyp_newclasses.loadPedigree(options)
>>> # Muck around with the database
>>> pyp_nrm.inbreeding(example)
>>> # Drop the existing table, if there is one
>>> pyp_db.deleteTable(example)
>>> # Check to see if the table is gone
>>> pyp_db.doesTableExist(example)
>>> # Creating the table
>>> pyp_db.createPedigreeTable(example)
>>> # Populating the table
>>> pyp_db.populatePedigreeTable(example)
>>> # Perform some calculations, in this case computing coefficients of inbreeding
>>> mean_inbreeding = pyp_reports.meanMetricBy(example,metric='fa',byvar='by')
>>> print mean_inbreeding

Which should give you the result:

[INFO]: Logfile hartlandclark.log instantiated.
[INFO]: Preprocessing hartlandclark.ped
[INFO]: Opening pedigree file hartlandclark.ped
[INFO]: Creating pedigree metadata object
        [INFO]:  Instantiating a new PedigreeMetadata() object...
        [INFO]:  Naming the Pedigree()...
        [INFO]:  Assigning a filename...
        [INFO]:  Attaching a pedigree...
        [INFO]:  Setting the pedcode...
        [INFO]:  Counting the number of animals in the pedigree...
        [INFO]:  Counting and finding unique sires...
        [INFO]:  Counting and finding unique dams...
        [INFO]:  Setting renumbered flag...
        [INFO]:  Counting and finding unique generations...
        [INFO]:  Counting and finding unique birthyears...
        [INFO]:  Counting and finding unique founders...
        [INFO]:  Counting and finding unique herds...
        [INFO]:  Detaching pedigree...
Metadata for Pedigree from van Noordwijck and Scharloo (1981) (hartlandclark.ped)
        Records:                15
        Unique Sires:         9
        Unique Dams:        5
        Unique Gens:         1
        Unique Years:         8
        Unique Founders:   3
        Unique Herds:         1
        Pedigree Code:       asdb
{1920: 0.0, 1960: 0.015625, 1930: 0.0, 1900: 0.0, 1970: 0.14453125, \
1940: 0.015625, 1910: 0.0, 1950: 0.078125}

Pretty snazzy, eh?

I’ve got no quarrel with SQLAlchemy — it looks very cool — but I could never quite get it to do what I wanted. After working with it for a day or so I concluded that it was just overkill for the few simple things I wanted to do, so I jumped ship to ADOdb for Python. It stood me in good stead many times in my PHP days,and it got the job done handily here, too.

I’m sure I’ll think about more that I want to say later. The dusting and polishing that goes along with preparing a release version is doing PyPedal a lot of good.

Now that I’ve established that my blood sugar’s off we can get down to brass tacks. Release candidates 1 and 2 have been released, and release candidate 3 is in the works. I won’t repeat everything that’s in the CHANGES.txt file, but I’ll hit the high points.

The big news is that I broke pyp_nrm/inbreeding_vanraden() at some point in the recent past. If I had regression tests I would have noticed this sooner. Maybe. Someone really should get on that. The bug affected only animals with both parents unknown or with unknown sires. The fix was very simple. I also fixed a bug that prevented the super-nifty Full-Sib Speedup from working. Wait, what, you’re not familiar with the Full-Sib Speedup? Let me explain.

The Full-Sib Speedup is a trick to avoid unnecessary calculations when you’ve got a pedigree with lots of full-sibs, such as dogs. The trick is that a list of observed sire-dam combinations is stored. The first time that combination is seen the resulting COI is stored in a dictionary. Any subsequent offspring of that combination receive the COI stored in the dictionary. Instead of calculating the COI for each offspring of the combination it is calculated only once. There are a number of steps avoided with this trick, such as pedigree extraction, reordering, and renumbering. And it now works correctly, which is a bonus.

The minor things include such stuff as lots of small fixes to the NewAMatrix class, fixes to inbreeding and relationship metadata, and added pedigree import/export from database and textstreams.

I also added a couple of chapters to the manual, one on input/output and one on performing calculations with pedigrees. The API documentation has been pulled out of the manual and is available on the website. I’m also now using Doxygen to generate those documents, which is working quite well.

Perhaps feature deletions are not as interesting as feature additions, but I’ve removed the GUI from PyPedal. The code that was hanging around was very old — it predated the current pedigree format system and used static format codes embedded in the pedigree file — and didn’t work very well. The fundamental idea behind PyPedal is that you get a set of tools that allow you to program (or metaprogram?) using pedigrees, and the GUI doesn’t fit in that paradigm. I’m also not comfortable with GUI programming and don’t have much interest in pursuing it. If someone wants to work on a GUI for PyPedal I’ll be happy to provide feedback, but do not intend to pursue it any further on my own.

Although Suwanlee et la. (2007) make a strong case that Ballou’s (1997) equation needs to be modified because it assumes that “regular” inbreeding and ancestral inbreeding are independent, Ballou’s method has the advantage of being much faster to calculate. If the overestimation observed under Ballou’s method is acceptable then performance-for-accuracy may be an acceptable trade-off. Both methods are provided in pyp_metrics so that the user may select whichever best suits their needs.

Several other small changes have been made to PyPedal. The most important of these is that several of the methods in the class pyp_newclasses/PedigreeMetadata now use Python sets, which were introduced in Python 2.4. As a result, PyPedal will not run on Python versions earlier than 2.4.