Maybe I should add Sweave output to PyPedal…

First, I do not have plans for a Python 3 port at the moment. One of the big blockers on that is matplotlib, but it is being ported. I’m not yet sure about the status of ReportLab and the rest of PyPedal’s dependencies.

Now, for the more exciting stuff, which will be in the 2.1 release. I’ve started working out the algorithms for some of the new tools I want to add, such as pedigree merging. Ideally, I can figure out how to do this with the __add__() method so that the syntax is easy: think “ped3 = ped1 + ped2″. I also have some ideas for what “subtraction” and “multiplication” of pedigrees might mean, but they’re a little sketchier right now. Ideally, I need a way to get a publication about this stuff to make it worth my very scarce time right now.

I plan to implement a faster algorithm that the recursive tabular method of VanRaden for calculating coefficients of inbreeding, but I’m not yet sure which one.

Just this morning I had a breakthrough on unknown parent groups, so look for that, too.

That’s enough to keep me pretty for a while, particularly if I am going to test everything reasonably well.

Is there anything you want that I’m leaving out?

Happy New Year!

While I don’t know of many people using PyPedal I have benefited from the feedback of several users over the past several years, particularly Matthew Kelly, Dan Cieslak, and Matthieu Authier. A number of interesting features have been added because of their input, and some nasty bugs fixed, as well. I hope that people will continue to contribute bug report and feature requests.

So what’s next for PyPedal? Well, there will certainly be 2.0.x maintenance releases for bugfix purposes. There also will be a 2.1 development branch, which will focus on (in no particular order):

• Some novel tools for manipulating pedigrees — extracting individuals or groups of animals, merging pedigrees, that kid of thing.
• Better visualization tools, particularly for large pedigrees.
• Improved documentation, particularly in the code itself.
• Performance improvements, particularly in the inbreeding routines.

I guess that’s all I have to say for now. If you’re one of my users, thanks very much! If you’re not one of my users, there’s no time like the present to start using PyPedal.

• Change the processing of ASD pedigrees to use hashlib instead of the md5 module. This may result in user-selectable hash functions in the event that you have trouble with collisions.
• For some reason, the logfile is not being created properly. That needs fixed.
• The NewPedigree::save() method is going to get a new argument so that you can specify the format of the output file.

So, okay, less talking and more coding.

On September 1, 2010, I uploaded Release Candidate 9 and let it go into the wild. In addition to adding GENES 1.20 input and output, the changes include some minor documentation updates and a few small code fixes.

In Release Candidate 10, which will be released this week, all I’ve done is add an option to pyp_nrm.fast_a_matrix() to add coefficients of inbreeding read from the pedigree file to the diagonal elements for founders when forming the relationship matrix. I need to update the documentation and check the files once more because I think that there are still some stray hard-coded debugging messages which need to be removed.

The only other thing I’m thinking of adding is support for unknown parent groups, and possibly adding methods to the NewPedigree class that support adding (merging) and subtracting (splitting) pedigrees, but I think that’s going to get punted to the 2.1 development branch because it’s a much harder problem than it may sound on the surface.

I still need bug reports and feature requests. I’ll do my best to fix bugs promptly. I can’t  make any promises about feature requests. My thinking right now is that I need to focus on polishing things. The demographics (pyp_demog), graphics (pyp_graphics) and reporting (pyp_reports) modules all could use a lot of fine-tuning to make them more useful. The documentation always needs work. I’m still concerned about the implementation of the algorithms in pyp_metrics for calculating effective ancestor numbers. They’re much less fun that implementing new features, but probably necessary. And then there’s the whole issue of dense SNP data…

These errors, in a way, exposed the tip of an iceberg: there is not nearly enough unit testing in PyPedal. Many of the example programs were written a long time ago (about three years) and not checked systematically since. This means that some programs, such as examples/new_amatrix.py, were still using methods that no longer exist (I’m thinking specifically about the info() method of NewAMatrix objects). Heck, that program still imported Numeric! So it’s a great idea to have examples, but it doesn’t help users when they run the program and everything fails. It makes the whole package look shoddy and undependable, and who wants to trust their work to that kind of code? If Matthieu hadn’t e-mailed me about those problems I’d still be shipping broken code. I hate writing unit tests as much as the next guy, but PyPedal’s grown to the point that it’s too big for me to keep track of, so I need a way to automate testing. So, it’s getting to be unit test time. With that and the work I want to do on the graphics module there’s plenty of work in the queue for version 2.0.1.