Some experiments I have been doing with workflow over the past semester as introduced on this post back in January.
1. Using Pandoc, Markdown, and pweave for class preparation.
I did the notes for my Logistics and Supply Chain course completely in Markdown. This included slides for in class presentation, writing homework solutions, and everything else with the class preparation. I wrote a Makefile that took care of everything. Basically what it meant is that to create slides for class prep I wrote it as a text file. It does force some simplicity because there is no ability to tweak a slide or an image. But that is probably a good thing because it forces efficiency.
One nice advantage of using pweave for homework and exam solutions is that it makes it easy to have multiple versions of an exam. If I have equations and steps in place, I can change a few parameters, and maybe an assumption or two and I have a new problem with not much extra work. Also I did a lot of cut and pasting of code. The code I wrote for presenting in class became the code that I used to solve the homework. And since I let them see the code (and I also copied over the equations needed to solve the homework) it connected the homework solution to the lectures. One issue is that so few of my students were competent programmers in any language that the Python code was not all that helpful to them (other than proving that I did indeed solve the problem) But this was a definite success.
2. Using Mercurial as a hosted version control system.
I had to work with a new team on one project I was working on, so the first thing I did was to move the former post-doc's work into a version control (Mercurial) and put it on a hosted repository (bitbucket). And we spent a month refactoring the entire thing to separate out the code from the site specific bits (because we were going to apply it to a new location). The repository meant that we were able to be aggressive in refactoring (yes, I did have to completely through out a day of work at one point because what I was thinking was not going to work. All I did was wipe my directory and reclone the repository) The others got into the hang of putting everything in the repository as well. Now, the comments are not all that good on the part of the post-doc and grad student working with me, and we tended to work in separate areas so it was not true collaborative programming, but it was not bad.
I plan on continuing this over the summer with a few students who are learning agent based modeling together. The intent is that if all of us are putting our models in the repository, we can ask questions like "why does ___ not work" and we can have their model to look at. We'll see how that works.
3. Google Tasks as a task organizer
This worked out pretty well. I ended up using Google tasks as a note taker (more on that later) I was off an on regarding being consistent, but there were periods where I had a lot going on that this to-do list being available on all of my computers, my iphone and ipad was very useful. Keep doing this.
4. Tomboy as a notes organizer.
Note taking sounds like a no-brainer. Not. This one pretty much died. In February Canonical eliminated an API that enabled use of Ubuntu One as a storage place. So I ended up storing the notes data on Dropbox (so my Windows and Linux machines were synced). But the iPad app that integrated with Tomboy no longer worked, so if I was using the iPad, the notes were not integrated. And this pretty much meant I was not using Tomboy notes. Sometimes I would create a task in Google tasks and create a note if I needed something. But generally this went unused.
I'm going to try this again. But since I realize now that having it on my iPad and syncing to my computers is the key to the whole affair, my main focus was on the iPad integration. So I am looking at Simplenote for the note-taking. It is originally designed for iPhone/iPad, and there are applications on Windows and Linux that integrate with it (and also the Google Chrome browser)
Showing posts with label version control. Show all posts
Showing posts with label version control. Show all posts
Wednesday, May 01, 2013
Sunday, January 06, 2013
Personal experiments on workflow, note takers and task organizers
This semester I'm undertaking a few experiments in productivity. Now, most of my work as a professor does involve computers, either developing and implementing models and methodology, preparing course lectures and materials, and communicating, and managing others. And this spring, I have a fairly challenging schedule from an organizational standpoint. In addition to the standard course and research projects I am working with a new (to me) graduate student and post-doc, I am running the department senior project course (with the department chair to back me up), and I am involved in one of those long-term career milestones. So part of this past winter break was spent setting up systems to improve productivity. Some minor workflow issues. And some larger ones.
1. Using Pandoc, Markdown, and pweave for class preparation.
I long ago decided on Beamer as my preferred method for preparing presentations. Using LaTeX essentially forces the use of styles instead of fiddling with page layout (or having your styles get overridden without notice) and I can focus on content rather than form. But there is alot of scaffolding that goes into any LaTeX document. Enter Markdown. I already use Markdown with R using knitr, which allows me to keep both the documentation about a method along with its coded implementation and the output of the code in a single document with a syntax that is not that far off what you would naturally do if you had to format a document using plain text (e.g. using Underlining and blank lines to set off headings or '*' to identify emphasized words.) So Pandoc is a utility that allows me to use Markdown format to generate slides in HTML5 or as PDF files (via Beamer). And pweave does for Python what knitr and Sweave do for R, combine the documentation of algorithms, the implementation, and the output in a single document where the output will automatically update as the implementation or the input data changes.
So now, I can build slides in Markdown that includes the lecture, the math model, a Python implementation of the math model, and the output of the implementation. And since Markdown is a lot less complex than LaTeX, hopefully do this faster than I was before.
2. Using Mercurial as a hosted version control system.
I was running a project that was based on a simulation being built over a period of years by a post-doc. However, he was really the only one working on the code. It was at the point where I wanted to do some side experiments with our model, so I wrote another implementation in Python. (note, using SimPy, it took me two weeks). But now he has moved on, and I'm still working on the project with another post-doc and grad student. So, since they are new, and we are jointly working on the model to apply it to a new system, I've taken the model from Dropbox (shared folders) to distributed version control, namely Mercurial on Bitbucket.
While Git is the more popular (at least among hipster elite programmers), Mercurial uses commands similar to Subversion (which the post-doc already knew) and tools were readily and obviously available for both Windows and Linux.
I've been using Mercurial and Bitbucket for my own projects including coding and writing papers in LaTeX for a while now, but this is the first project of my own that I have other people working with the same repository. So I'm training them along the way.
3. Google Tasks as a task organizer
I've started using Google Tasks. At one point I was using Remember the Milk, but I stopped because it was not integrating well with Google Calendar (which is how I do my scheduling). (RTM put an entry into every calendar day, which means I had to open my calendar to see if the marker was real or empty)
So with Google tasks, I can create a task in the Google Calendar sidebar, in the sidebar of Google mail, or there is an iOS app (GoTasks). The interface is not as pretty as Remember the Milk, but it works better with my Calendar. And the iOS version is not crippled (RTM is limited to one update a day on the free version). The main criteria here was the fact that it worked on all of my computers (I use Windows, Mac OS X, and Linux in different settings) as well as my iPhone and iPad (because I often go to meetings and trips with just my iPad and not my laptop)
4. Tomboy as a notes organizer.
This past fall I was working with an undergraduate who learned the use of MS OneNote while a summer associate with McKinsey and I was impressed about how well he could organize and use these free form notes. I had been using Google Docs to do this, but the need to get on the internet, fire up the browser, and then work was sometimes too much, and it was hard to organize.
Tomboy is a fairly lightweight application. It provides a formattable and linkable text area, but not as heavy as most desktop applications. Tomboy can also work with a cloud storage account such as Dropbox or Ubuntu One to keep the data in the cloud, to be synced whenever your computer is on-line. So now, I have set up a notes folder for the senior design project, with subfolders for each team, a folder for my major research project (with the grad student and post-doc), and my other project. And I know have collections of short notes that are each dated, identified with keywords, and linked to each other as much as needed. And I can access and edit them from each of my computers, my iPhone and iPad.
So, here is the great experiment, cobbled together in the midst of all the other work and writing I was doing over the winter break. And next week is when the fun starts.
1. Using Pandoc, Markdown, and pweave for class preparation.
I long ago decided on Beamer as my preferred method for preparing presentations. Using LaTeX essentially forces the use of styles instead of fiddling with page layout (or having your styles get overridden without notice) and I can focus on content rather than form. But there is alot of scaffolding that goes into any LaTeX document. Enter Markdown. I already use Markdown with R using knitr, which allows me to keep both the documentation about a method along with its coded implementation and the output of the code in a single document with a syntax that is not that far off what you would naturally do if you had to format a document using plain text (e.g. using Underlining and blank lines to set off headings or '*' to identify emphasized words.) So Pandoc is a utility that allows me to use Markdown format to generate slides in HTML5 or as PDF files (via Beamer). And pweave does for Python what knitr and Sweave do for R, combine the documentation of algorithms, the implementation, and the output in a single document where the output will automatically update as the implementation or the input data changes.
So now, I can build slides in Markdown that includes the lecture, the math model, a Python implementation of the math model, and the output of the implementation. And since Markdown is a lot less complex than LaTeX, hopefully do this faster than I was before.
2. Using Mercurial as a hosted version control system.
I was running a project that was based on a simulation being built over a period of years by a post-doc. However, he was really the only one working on the code. It was at the point where I wanted to do some side experiments with our model, so I wrote another implementation in Python. (note, using SimPy, it took me two weeks). But now he has moved on, and I'm still working on the project with another post-doc and grad student. So, since they are new, and we are jointly working on the model to apply it to a new system, I've taken the model from Dropbox (shared folders) to distributed version control, namely Mercurial on Bitbucket.
While Git is the more popular (at least among hipster elite programmers), Mercurial uses commands similar to Subversion (which the post-doc already knew) and tools were readily and obviously available for both Windows and Linux.
I've been using Mercurial and Bitbucket for my own projects including coding and writing papers in LaTeX for a while now, but this is the first project of my own that I have other people working with the same repository. So I'm training them along the way.
3. Google Tasks as a task organizer
I've started using Google Tasks. At one point I was using Remember the Milk, but I stopped because it was not integrating well with Google Calendar (which is how I do my scheduling). (RTM put an entry into every calendar day, which means I had to open my calendar to see if the marker was real or empty)
So with Google tasks, I can create a task in the Google Calendar sidebar, in the sidebar of Google mail, or there is an iOS app (GoTasks). The interface is not as pretty as Remember the Milk, but it works better with my Calendar. And the iOS version is not crippled (RTM is limited to one update a day on the free version). The main criteria here was the fact that it worked on all of my computers (I use Windows, Mac OS X, and Linux in different settings) as well as my iPhone and iPad (because I often go to meetings and trips with just my iPad and not my laptop)
4. Tomboy as a notes organizer.
Tomboy is a fairly lightweight application. It provides a formattable and linkable text area, but not as heavy as most desktop applications. Tomboy can also work with a cloud storage account such as Dropbox or Ubuntu One to keep the data in the cloud, to be synced whenever your computer is on-line. So now, I have set up a notes folder for the senior design project, with subfolders for each team, a folder for my major research project (with the grad student and post-doc), and my other project. And I know have collections of short notes that are each dated, identified with keywords, and linked to each other as much as needed. And I can access and edit them from each of my computers, my iPhone and iPad.
So, here is the great experiment, cobbled together in the midst of all the other work and writing I was doing over the winter break. And next week is when the fun starts.
Thursday, July 22, 2010
Looking at source code version control: Comments wanted
I've been investigating version control systems for use by a team. Is there anyone who uses these who can have some comments? I'm leaning towards Mercurial, but I am open to change (our best programmer likes Git). What follows are my notes on some version control systems. Note: While I personally like/use Linux, as a team we are being provided with Windows machines for this project. Based on my review, I'm only looking at Subversion, Mercurial and Git. Comments and experience would be helpful.
=====
Version control systems (VCS) are used to track historical changes to source code (text) files as well as a means to enable collaboration between team members working on a single project. Note that this can be for any text based project, both programming or LaTeX writing. There are 2 major advantages:
- Rollback: It is possible to retrieve the version of a file at any point in time. This is especially useful when one is exploring a new idea that ends up not working (or has introduced a bug or error into a program/file) (Note: Apple Computing advertises this feature in recent versions of Mac OS X as the "Time Machine", which is essentially version control built into an operating system).
- Collaboration: In the case of multiple people working on a different parts of a single project, changes that are made by one person can be committed to a shared repository, then distributed to all other team members. This results in a definitive version of a system, even as multiple team members are involved in ongoing work on their own parts of the system. Is it does so, version control can keep track of the changes, so that if a change is determined to cause problems and needs to be removed, it can be removed for everyone (or only on an individual's local version as needed)
There are two major categories of modern version control system: Centralized and Distributed
Centralized version control requires the use of a server, and all team members can update or branch using the centralized repository. For distributedversion control, all team members have a local version of source code repository, that can be synchronized to a definitive repository as needed. Distributedversion control is a relatively recent development (
- Native availability under MS Windows (availability under UNIX/LINUX is true for almost all Version Control systems). Preferably without use of Cygwin or MSYS.
- Can be used through GUI - through the operating system file explorer or a stand alone GUI (all open source VCS enable command line use)
- Integration with IDE - Visual Studio, Eclipse etc (EMACS/VIM integration can be assumed for open source VCS)
- Documentation - Tutorials available for those learning the system with the expectation of infrequent use
- Existing support within ___
- Available for MS Windows, Linux, Mac
- Standard plugins for Visual Studio (http://www.visualsvn.com/visualsvn/) and Eclipse (http://subclipse.tigris.org/).
- TortoiseSVN (http://tortoisesvn.net/) is a standard gui that integrates into Windows Explorer (MS Windows) as a context sensitive menu (usually right-click on a file to see options)
- Requires server
- The O'Reilly Press Subversion book is available as a freely available ebook (http://svnbook.red-bean.com/)
- Many tutorials available
- Built on UNIX shell and Perl. For Windows, this seems to require that CYGWIN or MSYS be installed to provide UNIX shell services on Windows.
- Available for MS Windows, Linux, Mac
- Multiple plugins for Visual Studio (http://visualhg.codeplex.com/) and Eclipse (http://javaforge.com/project/HGE).
- TortoiseHg (http://mercurial.selenic.com/wiki/TortoiseHg) is a standard gui that integrates into Windows Explorer (MS Windows) or Nautilus (Linux file explorer) as a context sensitive menu (usually right-click on a file to see options)
- The O'Reilly Press Mercurial book is available as a freely available ebook
- Some tutorials available. http://hginit.com has an introduction to version control tutorial.
- Also http://mercurial.selenic.com/wiki/UnderstandingMercurial and a tutorial (http://mercurial.selenic.com/wiki/Tutorial) and a Quick Start guide (http://mercurial.selenic.com/wiki/QuickStart)
Evaluation
For evaluation purposes we will compare three systems Subversion, Git and Mercurial, each of which are open source (license allows for free distribution) and in widespread use including a number of high profile large projects.
Subversion - Centralized version control system http://subversion.apache.org/
Git - Decentralized version control system
Mercurial - Decentralized version control system
http://mercurial.selenic.com/
MS Visual Source Safe / MS Team Foundation Server / IBM Clear Case are commercial Centralized VCS systems that are often used. In corporate environments they are often mandated by upper management because of their connections with Microsoft and IBM. Comments from working programmers are near-universally derogatory. The complaints include too tight integration with other vendor tools (so if an environment is not in MS Visual Studio, it is very difficult to get it into VSS/TFS (e.g. LaTeX files) ) In addition, there are numerous complaints that the repositories are difficult to manage, to the extent that it seems to be very easy to corrupt a repository, (making it useless)
There are other open source and commercial VCS systems (e.g. CVS, Perforce, AccuRev, Bazaar). Subversion, Git and Mercurial were chosen as the comparison set because as a set they are consistently are viewed as superior to others in comparisons among VCS. Note that because distributed VCS is so new (2005), comparisons more then 2 years old should be considered obsolete.
Some references:
Overview of Version Control tools by Martin Fowler (writer and consultant on organizing programming teams)
IBM DeveloperWorks article on Introduction to Distributed Version Control Systems
Wednesday, November 07, 2007
Computer development platform Part 2
Last month I started working on a model development aspect of a project. And I figured I'd take the time to learn the use of a new toolkit for programming. An update.
I had a project meeting yesterday that went very well. The approach was sound. The work on the model that I've done already has uncovered issues that have not been addressed by prior work, and this is going to lead to major policy recommendations. All the things that a model developer hopes will happen.
In addition, discussion uncovered some details in the system being modeled that I did not realize (helps when multiple heads get together to tease details out). And we are talking deployment when all is said and done. So, I have a few issues:
Well, this whole thing just screams out, unit testing. As it turns out, I've separated the building block objects from the business logic portions and the database access, which makes the whole thing easier. So, I build my test suite in PyUnit that takes the specifications and builds the model. Then the fun starts.
A very productive day. I almost think that I'm getting the hang of this programming stuff.
I had a project meeting yesterday that went very well. The approach was sound. The work on the model that I've done already has uncovered issues that have not been addressed by prior work, and this is going to lead to major policy recommendations. All the things that a model developer hopes will happen.
In addition, discussion uncovered some details in the system being modeled that I did not realize (helps when multiple heads get together to tease details out). And we are talking deployment when all is said and done. So, I have a few issues:
- The business logic of the model needs to be modified to handle my improved understanding of the process.
- A number of additional scenarios need to be considered, preferably without breaking anything else.
- The whole thing needs to be deployable. And that usually means MS Excel, a Visual Basic application, or Java (because these are things that can be sent to any computer without much trouble)
Well, this whole thing just screams out, unit testing. As it turns out, I've separated the building block objects from the business logic portions and the database access, which makes the whole thing easier. So, I build my test suite in PyUnit that takes the specifications and builds the model. Then the fun starts.
- I was still working on the business logic that allocates resources. It was messy, but it did provide output. So I wrote test cases that tested discrete stages along the way and rewrote the logic into smaller functions.
- Converting to Jython broke some functions, because newer versions of Python had some features that were not in Jython (some forms of introspection). So, I had to rewrite these so they worked under the older language specification, without breaking my test cases.
- Next on the list, write test cases that implement some of the newly described scenarios, and make them work.
A very productive day. I almost think that I'm getting the hang of this programming stuff.
Wednesday, October 03, 2007
Computer development platform
Well, I've started modeling work on a project at work. So far it has been data gathering planning (what kind of data do we need) and asking lots of questions about the system I'm trying to model. And, while that is not over, now that I'm back in Pittsburgh, time to get started. Right now, I'm actually doing the "make the first one to throw away" thing. Because this is so amorphous, I'm not sure just how to solve this just yet anyway. I told the project manager this, and he actually likes the idea as a risk management principle. (because version 2 is supposed to be deployable, let's just forget about that until I have a model that works, then worry about making it deployable)
Now, a lot of this is the fact that I'm working in a domain that not too many modelers have gone in. Slides that I've built for presentations in the past have gotten real good reception, as people who have seen this get the idea, and the possible impact. Of course now I have to deliver.
Since I get to work on that most idealized of environments, clean slate and zero previous work to build on, I thought I would actually try to do things right. And take the opportunity to build up a new tool chest. For now, documentation is in LaTeX, with Dia to build up flow charts and UMLish diagrams. I'm reading Head First Design Patterns by Freeman and Freeman, and some ideas have gotten in my head.
For the model building, there is an idea in the back of my head that deployment will be inside MS Excel and Visual Basic (because everything quantitative is deployed inside MS Excel and Visual Basic). But because the tools are just hard to develop in, and it quickly turns unmanageable, even when done right, I'll probably use Python and/or R to develop the initial model. One big benefit is the various algorithm libraries that are readily available in Python and R as well as the automatic documentation tools like Docstrings, PyUnit and Sweave. And I've actually used all of them.
For the tool chain, LaTeX and Dia and Python. But, since I had to get a new laptop anyway, and needed to reinstall my tools, why not try something new. I'm doing almost everything inside of the Eclipse IDE, specifically the EasyEclipse version of it. In addition to Java development (which I may end up doing non of, unless I switch into Jython), it handles LaTeX (through TeXlipse), Python, through Pydev), database access, and version control through Subversion. First time I've used many of these tools before. It is rather nice. TeXlipse is much better then the JEdit or Vim that I've used in the past since it does command completion, and it seems even better then TeXnicCenter that I did my thesis in. Having version control as an integral part of the IDE is wonderful. I've tried it in the past and it just got too bulky, even with a graphic interface. Now it is just a button click. Auto-build in Eclipse is nice too. Having a window with the results updated with every save means near-realtime feedback. Since Dia saves diagrams as XML, that works in a version control system also. So all my notes, the documentation, diagrams and the source code will all be in version control. I almost feel like I know what I am doing with this development thing.
Of course, the real fun starts next. Going from all my diagrams to actual code and data.
Now, a lot of this is the fact that I'm working in a domain that not too many modelers have gone in. Slides that I've built for presentations in the past have gotten real good reception, as people who have seen this get the idea, and the possible impact. Of course now I have to deliver.
Since I get to work on that most idealized of environments, clean slate and zero previous work to build on, I thought I would actually try to do things right. And take the opportunity to build up a new tool chest. For now, documentation is in LaTeX, with Dia to build up flow charts and UMLish diagrams. I'm reading Head First Design Patterns by Freeman and Freeman, and some ideas have gotten in my head.
For the model building, there is an idea in the back of my head that deployment will be inside MS Excel and Visual Basic (because everything quantitative is deployed inside MS Excel and Visual Basic). But because the tools are just hard to develop in, and it quickly turns unmanageable, even when done right, I'll probably use Python and/or R to develop the initial model. One big benefit is the various algorithm libraries that are readily available in Python and R as well as the automatic documentation tools like Docstrings, PyUnit and Sweave. And I've actually used all of them.
For the tool chain, LaTeX and Dia and Python. But, since I had to get a new laptop anyway, and needed to reinstall my tools, why not try something new. I'm doing almost everything inside of the Eclipse IDE, specifically the EasyEclipse version of it. In addition to Java development (which I may end up doing non of, unless I switch into Jython), it handles LaTeX (through TeXlipse), Python, through Pydev), database access, and version control through Subversion. First time I've used many of these tools before. It is rather nice. TeXlipse is much better then the JEdit or Vim that I've used in the past since it does command completion, and it seems even better then TeXnicCenter that I did my thesis in. Having version control as an integral part of the IDE is wonderful. I've tried it in the past and it just got too bulky, even with a graphic interface. Now it is just a button click. Auto-build in Eclipse is nice too. Having a window with the results updated with every save means near-realtime feedback. Since Dia saves diagrams as XML, that works in a version control system also. So all my notes, the documentation, diagrams and the source code will all be in version control. I almost feel like I know what I am doing with this development thing.
Of course, the real fun starts next. Going from all my diagrams to actual code and data.
Subscribe to:
Posts (Atom)