Flask Web Development by Grinberg: Book Review

Flask Web Development: Developing Web Applications with Python by Miguel Grinberg
My rating: 4 of 5 stars

I'm not a web developer, but Flask has always had an appeal of potentially being a potential front end to a database based application. But I've never gotten very far as tutorials generally look at only the main application, and I'm pretty sure I need some extensions but I have not been able to assess the quality of the many plugins available. Flask Web Development is that tutorial that shows Flask and selects quality extensions to introduce.

Flask Web Development is written as a tutorial, not a reference. As such Grinberg can decide on what is important. It starts like many other Flask tutorials in looking at the basic application structure, templates and web forms. But then it looks at databases along with a set of extensions for database management built around SQLAlchemy: Flask-SQLAlchemy, Flask-migrate. (and email, which I don't do)

He brings it all together with walking you through a blogging application. What gives the tutorial an over-the-shoulder feel to it is an innovative use of the github repository that goes with the book. Instead of having source files in the repository, the repository uses tags to incrementally build source files, so it is really like working alongside someone who knows what they are doing as they build the application. One tag will have a basic working version of functionality, and checking out subsequent tags builds out more features.

Flask Web Development covers many aspects of web programming, well beyond what most Flask tutorials will cover. I appreciate the deep dives into database management with SQLAlchemy, and the sections on testing and profiling which have applications beyond web development.

Disclaimer: I received a free electronic copy of Flask Web Development as part of the Oreilly Blogger programming.



View all my reviews

High Performance Python By Micha Gorelick, Ian Ozsvald: Book review

For someone like me who is a technical programmer but did not study CS, I've seen hints on how to speed up Python numerical code, but I only had a vague understanding of the principles and application.  This Early Release version of High Performance Python has examples that demonstrate why certain data structures are faster than others in particular situations, and how to use the various data structures provided.

But what may be unique is the chapter on the ways of speeding up Python through compiled code. There are many ways of using compiled code through Python, C and FORTRAN extensions, Cython and PyPy, and more recently Numba. But this book explains the strengths and limitations for each, along with a number of other ways of using compiled code along with Python that I had not heard of before. There are many references for each of these, but no general overview of this group of resources.

This is still an Early Release stage, so there are some warts. Many of the code examples are raw, and you have to know what you are doing to fill them in and get them to work. A website or Git repository with the source code examples would be very helpful.

Disclaimer: I received a free electronic copy of the Early Release edition of this book as part of the O'Reilly Press Blogger Program.

Mining the Social Web, 2nd ed by Matthew Russell: Book review

Mining the Social Web: Data Mining Facebook, Twitter, LinkedIn, Google+, GitHub, and More by Matthew A. Russell
My rating: 4 of 5 stars

The hardest part of learning a data analysis method is not in implementing the method, it is applying the method in the context of a real data problem. And data mining and machine learning texts often skirt the issue by using pre-processed data sets and problems defined to fit the method being taught. Russell uses analysis of social media sites to set a context where you start from having to gain access to real data sets, clean and transform the data into forms that your analytical libraries can make sense of, and then use the results to make a conclusion. For that, it rates a place along any other text that focuses more on the analytical methodology itself.

What I most appreciated about this book was the work put into converting data from one format to another. From the beginning, when he works with data pulled using a services API, then getting that into a format that another library requires, then getting those results into a data mining framework for analysis. Following his flow has helped me understand the methods better. And these examples of processing data from format to format is something that gets my students stuck before they get really started in a project. I especially appreciated the chapters that worked with the Natural Language Toolkit (NLTK) and the NetworkX graph libraries. These examples helped me get pass what was the hard part for me in working with these libraries in previous encounters.

The virtual machine is also very helpful. I have always found the hardest part of working with Python for analytic computing has been teaching my collaborators how to get set up. And in data mining this is even harder than standard. I was able to get through his book installing everything on one machine, but on another I used the author's virtual machine, and I have pointed a student who was working with me to the virtual machine as well.

This is a great book to work through the mess of implementing data mining methods in real situations. It is not a theory book, but it serves its purpose well.

Note: I received a free electronic copy of this book from the O'Reilly Press Blogger program.

View all my reviews

Python Cookbook, 3rd edition by Beazley and Jones - Book review

Python Cookbook by David Beazley
My rating: 5 of 5 stars

The third edition of Python Cookbook is part of what seems to be a general trend of issuing new editions focusing on Python 3. For me, whose focus is on data analysis and technical computing, this is the time to be thinking about the change from Python 2 to Python 3 as the base libraries of Numpy, Scipy, Matplotlib, and iPython have been ported, and the various other libraries I use that depend on these are being ported as well. But this edition is not just porting the old cookbook, it is a complete rewrite to go with the big shift that was Python 2 to 3. Because a lot of what I used the old cookbook and many of the recipes at the ActiveState website was for handling issues related to crossing versions (I had some projects with Jython, which was several versions behind CPython) and ways of getting around issues that are purportedly solved in Python 3. For that the third edition of Python Cookbook fills its purpose of showing idiomatic ways of performing some programming tasks, and being a reference for how to do thing well and elegantly taking advantage of the language and libraries, not fighting it.

I focus on using Python as a data analysis platform. So I generally only learned as much of the language I needed to in order to use the scientific stack of Numpy, Scipy, Matplotlib, Pandas, and the libraries that were built around them. But that means that I have not gotten to know large portions of the standard library. And introductory books don't cover this either, they focus on using the language itself. There are also a large number of books and references that focus on Python as a web development tool or a system administration tool, so those parts of the standard library get a lot of coverage in teaching materials. But the rest you almost have to stumble upon. In ideal conditions, the way you would learn about much of the standard library is to have someone who was more experienced nearby show you what you needed to know, as she demonstrated methods in her code that did the things that you never learned in class. But sometimes there is no such person. The Python Cookbook plays this role, of demonstrating how to do things in Python 3 that are practical and you probably would not learn while learning the language itself.

Some areas that I found useful are the heapq, generators, and the I/O. heapq is a module and data structure I just never got around to learning. Usually discussions about Python data structures made their way to deque and heapq was discussed by reference. But after looking at the priority queue discussion, I fired up an iPython notebook and went through every recipe that used heapq and I've started thinking about how to rewrite a model I recently coded up. Generators and File I/O are areas that I knew in passing through my use with them in data analysis, but Python Cookbook opened up new ways of understanding (I am starting to get why JSON is so useful). Now, there is nothing special about these, but seeing these parts of the standard library in use in an elegant way is something beyond what you would get from a module documentation or a standard tutorial.

I do miss the introductions to each chapter that was in the 1st and 2nd editions of Python Cookbook. But what Python Cookbook does give is an idiomatic feel of using Python3, when there are not all that many mentors out there to go around. So this is something very useful for others who are starting to use Python 3. It is not for learning the language, but it is for using the language well.

Disclaimer: I received a free electronic copy of this book as part of the Oreilly Bloggers program

View all my reviews

A recap on work flow, note takers and task organizers

Some experiments I have been doing with workflow over the past semester as introduced on this post back in January. 

1.  Using Pandoc, Markdown, and pweave for class preparation.  

I did the notes for my Logistics and Supply Chain course completely in Markdown.  This included slides for in class presentation, writing homework solutions, and everything else with the class preparation.  I wrote a Makefile that took care of everything.  Basically what it meant is that to create slides for class prep I wrote it as a text file.  It does force some simplicity because there is no ability to tweak a slide or an image.  But that is probably a good thing because it forces efficiency.

One nice advantage of using pweave for homework and exam solutions is that it makes it easy to have multiple versions of an exam.  If I have equations and steps in place, I can change a few parameters, and maybe an assumption or two and I have a new problem with not much extra work.  Also I did a lot of cut and pasting of code.  The code I wrote for presenting in class became the code that I used to solve the homework.  And since I let them see the code (and I also copied over the equations needed to solve the homework) it connected the homework solution to the lectures.  One issue is that so few of my students were competent programmers in any language that the Python code was not all that helpful to them (other than proving that I did indeed solve the problem)  But this was a definite success.

2.  Using Mercurial as a hosted version control system.  

I had to work with a new team on one project I was working on, so the first thing I did was to move the  former post-doc's work into a version control (Mercurial) and put it on a hosted repository (bitbucket).  And we spent a month refactoring the entire thing to separate out the code from the site specific bits (because we were going to apply it to a new location).  The repository meant that we were able to be aggressive in refactoring (yes, I did have to completely through out a day of work at one point because what I was thinking was not going to work.  All I did was wipe my directory and reclone the repository)  The others got into the hang of putting everything in the repository as well.  Now, the comments are not all that good on the part of the post-doc and grad student working with me, and we tended to work in separate areas so it was not true collaborative programming, but it was not bad.

I plan on continuing this over the summer with a few students who are learning agent based modeling together.  The intent is that if all of us are putting our models in the repository, we can ask questions like "why does ___ not work" and we can have their model to look at.  We'll see how that works.

3.  Google Tasks as a task organizer  

This worked out pretty well.  I ended up using Google tasks as a note taker (more on that later)  I was off an on regarding being consistent, but there were periods where I had a lot going on that this to-do list being available on all of my computers, my iphone and ipad was very useful.  Keep doing this.

4.  Tomboy as a notes organizer. 

Note taking sounds like a no-brainer.  Not.  This one pretty much died.  In February Canonical eliminated an API that enabled use of Ubuntu One as a storage place.  So I ended up storing the notes data on Dropbox (so my Windows and Linux machines were synced).  But the iPad app that integrated with Tomboy no longer worked, so if I was using the iPad, the notes were not integrated.  And this pretty much meant I was not using Tomboy notes.  Sometimes I would create a task in Google tasks and create a note if I needed something.  But generally this went unused.

I'm going to try this again.  But since I realize now that having it on my iPad and syncing to my computers is the key to the whole affair, my main focus was on the iPad integration.  So I am looking at Simplenote for the note-taking. It is originally designed for iPhone/iPad, and there are applications on Windows and Linux that integrate with it (and also the Google Chrome browser)

Fernando Perez: "Literate computing" and computational reproducibi...

Fernando Perez: "Literate computing" and computational reproducibi...: As "software eats the world" and we become awash in the flood of quantitative information denoted by the "Big Data" b...


I am teaching a course in logistics and supply chain.  Because the course is focused on the modeling of supply chains, and the students are generally using Excel (I actually did all of the homework solutions in Python) I used the Reinhart and Rogoff’s “Growth in a Time of Debt” discussion as a mini-case (every class I find an article to discuss.  Usually it focuses on business decisions, but this time I chose this topic.)

The focus on the class discussion was on how you look for errors.  Because Excel makes this nearly impossible, the real question was on how you focus your time.

But the real answer, as Perez mentions, it to use tools that make reproducible research simple.  The homework assignments I gave resulted in spreadsheets that covered 4 tabs and were a couple hundred columns and thousands of rows.  My solution in python was about a page of commented code which had a near one to one correspondence with the mathematical formulation of the model (with the addition of a few lines to read and massage the data.)  The spreadsheets were pretty much un-auditable.  My code had comments that would tell a reviewer what to look for.  Even for smaller problems, my students spreadsheets were fairly obtuse while my Python programs (I use Pweave) alternated between the explanation of each step with the calculations described.

The Thomas Herndon, Michael Ash, and Robert Pollin paper included R code that both demonstrated the effects of the errors of Reinhart and Rogoff.

http://www.peri.umass.edu/236/hash/31e2ff374b6377b2ddec04deaa6388b1/publication/566/

For fun, Vincent Arel-Bundock has converted the Herndon, Ash, and Pollin into an iPython notebook, suitable for looking at (and reading) as well as downloading it and playing with the data to test any thoughts someone may have about testing the impact of various types of errors.

http://nbviewer.ipython.org/urls/raw.github.com/vincentarelbundock/Reinhart-Rogoff/master/reinhart-rogoff.ipynb

2012 Python meme

I think the official rules are here.  I don't have a Twitter feed, so I'll just put this tag in my labels.

• copy-paste the questions and answer them in your blog
• tweet it with #2012pythonmeme hashtag

1. What’s the coolest Python application, framework, or library you have discovered in 2012?

Hmm, I got to know alot about pandas this year, but I discovered it before.  So I have to say pweave.  I learned the whole concept of combining the narrative discussion, code, and output in literate programming through Sweave in R, and I love the idea of doing the same in Python.  Combined with Pandoc, this means I can write explanation, code, and results, have the data or code change, and my slides or documents are updated automatically.  And that makes a lot of things work better.

2. What new programming technique did you learn in 2012?
Using list comprehensions and other functional programming based patterns in data analysis.  Most of the programming I do tends to be data analysis (because the students and post-docs I work with are not skilled in this), and my code has been turning more functional over the year.

3. Which open source project did you contribute to the most in 2012? What did you do? 
Not much in the way of direct contributions.  I filled out bug reports with both pandas and R Studio.  I actually enjoyed the interactions I had with developers (both have developers employed by companies that are focused on these products) as we trouble shot the code and developed working solutions.

4. Which Python blog or website did you read the most in 2012?
I cheat and use Planet SciPy and get all of the scientific programming in Python blogs.

5. What are the top things you want to learn in 2013?
Working with both simulation libraries (SimPy) and optimization modeling languages (Pyomo or PuLP) and working them into a real problem.

6. What is the top software, application, or library you wish someone would write in 2013?  
Round trip between iPython Notebook and Python with Markdown.  Right now I think I have to choose between workflows, and there are some nice parts about each that I would like to keep.

I could say more Python 3 migration, but there are so many pieces that have to be moved, this would hardly result in a top N list for any reasonable value of N.

Personal experiments on workflow, note takers and task organizers

This semester I'm undertaking a few experiments in productivity.  Now, most of my work as a professor does involve computers, either developing and implementing models and methodology, preparing course lectures and materials, and communicating, and managing others.  And this spring, I have a fairly challenging schedule from an organizational standpoint.  In addition to the standard course and research projects I am working with a new (to me) graduate student and post-doc, I am running the department senior project course (with the department chair to back me up), and I am involved in one of those long-term career milestones.  So part of this past winter break was spent setting up systems to improve productivity.  Some minor workflow issues.  And some larger ones.

1.  Using Pandoc, Markdown, and pweave for class preparation.  

I long ago decided on Beamer as my preferred method for preparing presentations.  Using LaTeX essentially forces the use of styles instead of fiddling with page layout (or having your styles get overridden without notice) and I can focus on content rather than form.  But there is alot of scaffolding that goes into any LaTeX document.  Enter Markdown.  I already use Markdown with R using knitr, which allows me to keep both the documentation about a method along with its coded implementation and the output of the code in a single document with a syntax that is not that far off what you would naturally do if you had to format a document using plain text (e.g. using Underlining and blank lines to set off headings or '*' to identify emphasized words.)  So Pandoc is a utility that allows me to use Markdown format to generate slides in HTML5 or as PDF files (via Beamer).  And pweave does for Python what knitr and Sweave do for R, combine the documentation of algorithms, the implementation, and the output in a single document where the output will automatically update as the implementation or the input data changes.

So now, I can build slides in Markdown that includes the lecture, the math model, a Python implementation of the math model, and the output of the implementation.  And since Markdown is a lot less complex than LaTeX, hopefully do this faster than I was before.

2.  Using Mercurial as a hosted version control system.  

I was running a project that was based on a simulation being built over a period of years by a post-doc.  However, he was really the only one working on the code.  It was at the point where I wanted to do some side experiments with our model, so I wrote another implementation in Python. (note, using SimPy, it took me two weeks).  But now he has moved on, and I'm still working on the project with another post-doc and grad student. So, since they are new, and we are jointly working on the model to apply it to a new system, I've taken the model from Dropbox (shared folders) to distributed version control, namely Mercurial on Bitbucket.

While Git is the more popular (at least among hipster elite programmers), Mercurial uses commands similar to Subversion (which the post-doc already knew) and tools were readily and obviously available for both Windows and Linux.

I've been using Mercurial and Bitbucket for my own projects including coding and writing papers in LaTeX for a while now, but this is the first project of my own that I have other people working with the same repository.  So I'm training them along the way.

3.  Google Tasks as a task organizer

I've started using Google Tasks. At one point I was using Remember the Milk, but I stopped because it was not integrating well with Google Calendar (which is how I do my scheduling). (RTM put an entry into every calendar day, which means I had to open my calendar to see if the marker was real or empty)

So with Google tasks, I can create a task in the Google Calendar sidebar, in the sidebar of Google mail, or there is an iOS app (GoTasks). The interface is not as pretty as Remember the Milk, but it works better with my Calendar. And the iOS version is not crippled (RTM is limited to one update a day on the free version).  The main criteria here was the fact that it worked on all of my computers (I use Windows, Mac OS X, and Linux in different settings) as well as my iPhone and iPad (because I often go to meetings and trips with just my iPad and not my laptop)

4.  Tomboy as a notes organizer.

This past fall I was working with an undergraduate who learned the use of MS OneNote while a summer associate with McKinsey and I was impressed about how well he could organize and use these free form notes. I had been using Google Docs to do this, but the need to get on the internet, fire up the browser, and then work was sometimes too much, and it was hard to organize.

Tomboy is a fairly lightweight application.  It provides a formattable and linkable text area, but not as heavy as most desktop applications.  Tomboy can also work with a cloud storage account such as Dropbox or Ubuntu One to keep the data in the cloud, to be synced whenever your computer is on-line.  So now, I have set up a notes folder for the senior design project, with subfolders for each team, a folder for my major research project (with the grad student and post-doc), and my other project.  And I know have collections of short notes that are each dated, identified with keywords, and linked to each other as much as needed.  And I can access and edit them from each of my computers, my iPhone and iPad.

So, here is the great experiment, cobbled together in the midst of all the other work and writing I was doing over the winter break.  And next week is when the fun starts.

Book Review: Python for Data Analysis by Wes McKinney

Python for Data Analysis by Wes McKinney
My rating: 5 of 5 stars

For some time now I have been using R and Python for data analysis. And I have long ago discovered the Python technical stack of ipython, NumPy, Scipy, and Matplotlib and I thought I knew what I was doing. I even dipped my toe into pandas as my data structure for analysis. But Python for Data Analysis showed me entire worlds of improvement in my workflow and my ability to work with data in the messy form that is found in the real world.

Python, like most interpreted languages, is slow compared to compiled languages. But there is a technical stack that started with the NumPy libraries and has grown to include Scipy, Matplotlib (graphing), ipython (shell) and pandas you get high quality and fast algorithm and data structure Fortran and C libraries underneath Python. But while these libraries are designed to be used together, documentation tends to be only about one at a time, and very little puts it all together as an integrated whole. McKinney's Python for Data Analysis fills that gap.

Even though I have been using iPython, NumPy, Scipy and Matplotlib for years, and pandas for about half a year, going through this book makes me feel like I was a rank novice. I learned how to efficiently use the shell as a development tool, to the point I have stopped automatically using the ipython notebook or pydev (eclipse) when starting new projects and I use the shell instead, because its introspection and debugging capabilities made it much easier to work. I had started using pandas for a data structure because I liked the similarities with R data frames, this book showed me where pandas goes well beyond that. With matplotlib I could make specific plots, this book showed me how to use the pandas interface to make them a natural part of the workflow (even if it is not yet at the level of a grammer such as ggplots)

Python for Data Analysis does not just teach how to use the Python scientific stack, it also teaches a workflow for technical computing. And this is beyond what you can get from reading off the web, it probably really requires the opportunity to work alongside someone who knows what they are doing to see the practices that makes them productive. As such, I would recommend it for anyone who does scientific and technical computing, whether in the sciences, engineering, finance, or other areas where quantitative computing using Python is done.

Disclaimer: I received a free electronic copy of this book from the O'Reilly Blogger Program.

View all my reviews

Getting SageTeX to work on Ubuntu

Since I'm using Sage more, I'm trying to get SageTeX working, so I can embed Sage into LaTeX documents (i.e. have a report on the methods integrated with the code that implements the methods and generates the results. Search for 'Literate Programming' or 'Sweave' for many references on this)

On Ubuntu, after I copied SageTeX files to my local LaTeX configuration directory (as per instructions in the Sage Installation Guide, I still got errors when trying to build the example documents

Traceback (most recent call last):
File "st_example.py", line 7, in
_st_.inline(_sage_const_0 , number_of_partitions(_sage_const_5 ))
File "/home/lluang/Apps/sage-4.7.2/local/lib/python2.6/site-packages/sagetex.py", line 86, in inline
'}{{%\n' + s.rstrip() + '}{}{}{}{}}\n')
File "element.pyx", line 331, in sage.structure.element.Element.__getattr__ (sage/structure/element.c:2868)
File "parent.pyx", line 327, in sage.structure.parent.getattr_from_other_class (sage/structure/parent.c:3193)
AttributeError: 'sage.rings.integer.Integer' object has no attribute 'rstrip'

**** Running Sage on st_example.sage failed! Fix st_example.tex and try again.


So what happened? Apparently, one of the Ubuntu texlive packages (specifically texlive-latex-extra) includes an outdated version of the sagetex.sty file while what is needed is one tailored to the current version of Sage (and included in the Sage distribution). While I have the correct version in a local directory, there is a precedence issue when multiple LaTeX style files are present. Since I don't have the inclination to figure out the issue, I found the location of the wrong sagetex.sty file, and deleted the entire directory (/usr/share/texmf-texlive/tex/latex/sagetex). Now, SageTeX works, and I can start writing papers using Sage/Python, just like I do using Sweave for R.

Book Review: Think Stats by Allen Downey

Think Stats by Allen B. Downey
My rating: 4 of 5 stars

Statistics gets a little respect in Operations research, in part because it gets taught as a bunch of formulas and computer procedures. And the problem with the way that it is taught is that the formulas don't mean anything, and the student may know her way around menus, but that does not mean that she knows under what circumstances to use what method. And everything is learned in isolation, often without practice in getting her hands dirty. Think Stats gives students the chance to get their hands dirty.

Because it uses a programming language (Python) it covers data analysis from beginning to end: viewing data, calculating descriptive statistics, identifying outliers, describing data using the distributions (and explaining what the distributions really mean!). Going through this small book, the goal is understanding and using statistics, not just learning statistics. I have a number of college undergraduate students working on projects. I have started giving them this to work on when they first start with me, both for the programming in Python and to learn statistics and data analysis so they can be useful.

I received a free electronic copy of Think Stats from the O'Reilly Blogger review program.

View all my reviews

Review: Sage: Beginner's Guide by Craig Finch

Sage Beginner's Guide by Craig Finch

My rating: 4 of 5 stars


I like to use Python for modeling and data analysis, and I tell my students that I consider Matlab, R and Python moral equivalents, made in kind by their wrapping of various numerical Fortran libraries, data structures for matrices and vectors, and numerous specialized libraries. But while there are Matlab books for every combination of field and level, and R books for every branch of statistics under the sun, Python books for data analysis are rare. Most introduction books are aimed at computer administrators or web programmers. Material on the web for scientists tended to be reference material that explained the functions available. The few in depth books seemed to assume that you were already a competent scientific programmer who was adding a new language to the toolkit. Sage: Beginner's Guide is meant for the person who is learning scientific programming, and doing so using Sage. As such it is highly useful for those who are being introduced to scientific computing in Python world.

While I use Sage and Python in technical programming myself, I have not been able to successfully teach someone else to do the same. What Finch does is to introduce someone not only to tools available for Python programmers, but instructions on setting up the environment, the practice of technical programming, but also the idea that each of these steps sets up something else.

Sage is a large and highly capable program, so any book has to focus somewhere. So the chapters can be thought of as covering the following (Note: this is NOT a chapter listing):

• Introduction and installation of Sage


• Use of Sage as an interactive environment


• Python programming: Introductory and advanced programming


• Numerical methods: Linear algebra, solving equations, numeric integration, ODE


• Symbolic math: algebra and calculus


• Plotting: 2D and 3D

In each substantive chapter, topics are covered in a standard pattern. A brief narrative description, a short sample program that uses the concept, a description of what program does and why the output looks like it does, then sometimes there are exercises that you can use to confirm you understand the concept or build your intuitive understanding.

What is missing? These are probably additional topics for "Where do we go from here" chapter. First, they do not take advantage of the Python ecosystem. Because of the basics of Numpy, Scipy and Matplotlib, numerous other scientific libraries exist that are not in Sage. I would include some notes on installing packages for use in Sage (which requires some modifications to the standard procedures). Also, an explicit mention of Scipy, since it is the basis for a number of other scientific packages in Python.

Sage: Beginner's Guide is a great addition to the library. It fills the role of the introduction to technical programming in Python that for Matlab is filled by professors who teach computational science/engineering courses. I envision that my copy of the book will be loaned out to one student after another for some time to come.



View all my reviews

Note: I received a free copy of Sage Beginner's Guide for review from Packt Publishing.

Installing pyspatialite

Goal: To use spatial databases to perform operations such as centroid, nearest point, and distances to generate a table of distances along a road network from area to area, using the nearest node to the centroid of an area to represent the area.

Backup plan: Create a list of nearest node to centroid of areas. Use some code I have previously written in R to implement Dijkstra's algorithm.

Preferred method: Programmatically access Spatialite and perform all functions in spatialite.

Barrier: Most distributions of SQLite (including the sqlite3 library included in Python) do not enable loading of extentions.

Solution: Install Python module pyspatialite.

Installing pyspatialite is done through easy_install: sudo python setup.py install

The documentation mentions that you need geos and proj libraries installed. What it does not tell you is that you need the development versions (which include header files) of these libraries, as well as the Python headers. After installing the developmental files, then you can install pyspatialite.

Installing rSymPy on Ubuntu. JDK problem (and soln)

I've started using rSymPy, because I find myself having to being able to take derivatives and integrals again. And maybe a Computer Algebra System is something worth learning. I'm learning both Sage as well as Sympy.

Sympy has the advantage of running in pure Python. And with the recent version of Jython 2.5, it means Sympy can run on the Java Virtual Machine. And someone figured out that meant that you could call Sympy from R, using the rJava interface, using the rSymPy package

The problem is configuring R to use rJava. This can be tricky, because R has to be configured to use the Java Development Kit. Under Ubuntu, I have two machines that I had to put this on. It turns out this was easy for one machine, but difficult for the other.

The instructions are to configure R by running the following as root (or sudo)

R CMD javareconf

But then when installing rJava, I got the following error:

configure: error: One or more Java configuration variables are not set.
Make sure R is configured with full Java support (including JDK).

The problem was the JDK. When running javareconf I see:

Java home path : /usr/lib/jvm/java-6-openjdk/jre

Hmm, I thought I was using the Sun Java JDK. So I uninstall openjdk and reinstall the Sun provided JDK. After this, I run javareconf and get what I expect:

Java home path : /usr/lib/jvm/java-6-sun-1.6.0.15/jre

Now, when I use install.packages() to install rSymPy, it works.

Eclipse and 'TODO'

I can be so easily impressed sometimes. I was working on a model, and basically looking over some stuff I had put together in a couple of days to incorporate some feedback before a presentation. So it is a bit rough. And it is a bit late, so what I'm really doing is putting in a bunch of comments about what some of the code actually does and some things that I should do after I get some more sleep into me. And as I go along, I suddenly noticed that my screen was changing, other then just the fact that I was typing in code.

There was this little box to the left of the code.

So I was trying to figure out what was going on, because then I noticed I had a couple of these along with my debugging breakpoints. Then I figured it out. Among my many notes-to-self, I had put in some comments # TODO. And Eclipse was catching this and automatically building a task list for me.

Just a random note on the day. :-)

Open source develop: assertAlmostEqual

I've been working on a project at work while using Python/Jython. And I'm using unittesting to drive and test the development along the way. Because I deal with things like real data, the usual assertEqual runs into problems when you use floating point computation, because you just don't test for equality in floating point. To much potential for rounding errors at the 20th digit or some other insignificant problem. So you need to use assertAlmostEqual.

And everything is working fine, until I try to run my tests in Jython. And then all my floating point tests start to fail, because assertAlmostEqual does not exist. So, did I spell it wrong, but it worked under normal Python. So a quick google search reveals that assertAlmostEqual was added a bit later, so it was not in Python 2.2 (which Jython is written against). And there is a newsgroup posting in the Python-checkins list about the adding of assertAlmostEqual to the PyUnit testing library (which is what I use for my unit tests).

http://mail.python.org/pipermail/python-checkins/2002-December/032079.html

Sat, 28 Dec 2002 22:11:50 -0600

[Python-checkins] python/nondist/sandbox/twister test_random.py,1.1,1.2

Raymond> To accomodate single precision platforms, only test to seven
    Raymond> digits.

The SciPy folks have added an assertAlmostEqual method to their unit tests.
I believe it more-or-less just wraps what you've done in a callable method
(which takes a number of digits of precision).  Might be a good idea to add
something like it to unittest.py so the wheel doesn't keep getting
reinvented.  They actually have a few variants, coded as functions here:

http://www.scipy.org/site_content/remap?rmurl=http%3A//scipy.net/cgi-bin/viewcvsx.cgi/scipy/scipy_test/testing.py

Skip

Ok. That explains that. But something else looks oddly familiar. I remember doing something with SciPy around the same time
.

http://projects.scipy.org/pipermail/scipy-user/2002-January/008545.html

Sun Jan 20 21:38:03 CST 2002
[SciPy-user] unittests for scipy.stats: assert_almost_equal question

Hmmm, I think that I would like to be testing in terms of
significant digits as opposed to decimal places, especially when
working with floating point.  Since you are asking for such a test,
here it is.  The attached file has a function meant to go into the
scipy_test.py module.  I wrote assert_approx_equal following the
same form as assert_almost_equal
assert_approx_equal(actual,desired,significant=7,err_msg='',verbose=1):
compares 'actual' and 'desired' and determines the first
'significant' significant digits and checks for accuracy.  Let's
see, I think I counted significant digits correctly.  Can anyone
tell?

Louis
> From: "eric"
> To: <scipy-user at scipy.org>
> Subject: Re: [SciPy-user] unittests for scipy.stats:
> assert_almost_equal question
> Date: Sun, 20 Jan 2002 05:36:31 -0500
> Organization: enthought
> Reply-To: scipy-user at scipy.net
>
> Hey Louis,
> 
> A thousand blessing upon you.  I immediately commited it to the
> CVS!
> 

Oh, now I know why this problem looks so familiar. Way back when I was a grad student, I was trying to test some functions. And I ran into the problem about unittesting floating point. So I wrote unittest code to test floating point in SciPy. And it was added to SciPy. And the main Python language developers noticed (since those folks looked to the SciPy/Numpy folks about all things numerical computation related) and added it to the main language. And here I am, five years later, taking advantage of something I did as a grad student.

Add to reasons for "why contribute to open source programming" story.

Computer development platform Part 2

Last month I started working on a model development aspect of a project. And I figured I'd take the time to learn the use of a new toolkit for programming. An update.

I had a project meeting yesterday that went very well. The approach was sound. The work on the model that I've done already has uncovered issues that have not been addressed by prior work, and this is going to lead to major policy recommendations. All the things that a model developer hopes will happen.

In addition, discussion uncovered some details in the system being modeled that I did not realize (helps when multiple heads get together to tease details out). And we are talking deployment when all is said and done. So, I have a few issues:

1. The business logic of the model needs to be modified to handle my improved understanding of the process.
2. A number of additional scenarios need to be considered, preferably without breaking anything else.
3. The whole thing needs to be deployable. And that usually means MS Excel, a Visual Basic application, or Java (because these are things that can be sent to any computer without much trouble)

Number 3 leads to an additional complication. I've been developing the whole thing in Python, using Eclipse. I'm going to decide that for deployment, I will convert the whole thing to Jython, since that makes it deployable on a Java Virtual Machine (and everyone has Java installed). But Python is at version 2.5, while Jython is at version 2.2, so it is a couple years behind.

Well, this whole thing just screams out, unit testing. As it turns out, I've separated the building block objects from the business logic portions and the database access, which makes the whole thing easier. So, I build my test suite in PyUnit that takes the specifications and builds the model. Then the fun starts.

1. I was still working on the business logic that allocates resources. It was messy, but it did provide output. So I wrote test cases that tested discrete stages along the way and rewrote the logic into smaller functions.
2. Converting to Jython broke some functions, because newer versions of Python had some features that were not in Jython (some forms of introspection). So, I had to rewrite these so they worked under the older language specification, without breaking my test cases.
3. Next on the list, write test cases that implement some of the newly described scenarios, and make them work.

Some of this involved serious work. In particular, rewriting the code to version 2.2 vs. 2.5. Right before I did this, I did a commit to version control (Subversion) just in case I completely messed up. The IDE also helps alot since it takes care of version control, checking Python and Jython, and running the unit tests.

A very productive day. I almost think that I'm getting the hang of this programming stuff.

Computer development platform

Well, I've started modeling work on a project at work. So far it has been data gathering planning (what kind of data do we need) and asking lots of questions about the system I'm trying to model. And, while that is not over, now that I'm back in Pittsburgh, time to get started. Right now, I'm actually doing the "make the first one to throw away" thing. Because this is so amorphous, I'm not sure just how to solve this just yet anyway. I told the project manager this, and he actually likes the idea as a risk management principle. (because version 2 is supposed to be deployable, let's just forget about that until I have a model that works, then worry about making it deployable)

Now, a lot of this is the fact that I'm working in a domain that not too many modelers have gone in. Slides that I've built for presentations in the past have gotten real good reception, as people who have seen this get the idea, and the possible impact. Of course now I have to deliver.

Since I get to work on that most idealized of environments, clean slate and zero previous work to build on, I thought I would actually try to do things right. And take the opportunity to build up a new tool chest. For now, documentation is in LaTeX, with Dia to build up flow charts and UMLish diagrams. I'm reading Head First Design Patterns by Freeman and Freeman, and some ideas have gotten in my head.

For the model building, there is an idea in the back of my head that deployment will be inside MS Excel and Visual Basic (because everything quantitative is deployed inside MS Excel and Visual Basic). But because the tools are just hard to develop in, and it quickly turns unmanageable, even when done right, I'll probably use Python and/or R to develop the initial model. One big benefit is the various algorithm libraries that are readily available in Python and R as well as the automatic documentation tools like Docstrings, PyUnit and Sweave. And I've actually used all of them.

For the tool chain, LaTeX and Dia and Python. But, since I had to get a new laptop anyway, and needed to reinstall my tools, why not try something new. I'm doing almost everything inside of the Eclipse IDE, specifically the EasyEclipse version of it. In addition to Java development (which I may end up doing non of, unless I switch into Jython), it handles LaTeX (through TeXlipse), Python, through Pydev), database access, and version control through Subversion. First time I've used many of these tools before. It is rather nice. TeXlipse is much better then the JEdit or Vim that I've used in the past since it does command completion, and it seems even better then TeXnicCenter that I did my thesis in. Having version control as an integral part of the IDE is wonderful. I've tried it in the past and it just got too bulky, even with a graphic interface. Now it is just a button click. Auto-build in Eclipse is nice too. Having a window with the results updated with every save means near-realtime feedback. Since Dia saves diagrams as XML, that works in a version control system also. So all my notes, the documentation, diagrams and the source code will all be in version control. I almost feel like I know what I am doing with this development thing.

Of course, the real fun starts next. Going from all my diagrams to actual code and data.