Teaching kids micro:bit and makecode during COVID-19 quarantine

A couple of weeks into social distancing, as the kids various activities had shut down with no plan on how to open, we were faced with the question of how to keep our kids occupied.  We have generally followed the principle of having one artistic, one physical, and one STEM activity ongoing per child (plus chinese), and for the older one we were trying to fill in the space opened as his robotics (First LEGO League) activity had closed down.  And we wanted to have it be somewhat social, as we also wanted him to maintain his connections, as well as make sure there was a community of friends and relationships that may endure.

For classmates, our first community that we reached out to were the other kids from his LEGO robotics group.  I already had a number of email addresses from organizing an outing, and these were all kids who knew blockly style programming (or Scratch like) and programming in the physical world.  The next source was a local chinese parent WeChat group.  And we were looking for kids roughly the same age as DS9.  And we found one more from a camp that DS9 was in last year.

The goal was teaching programming, and for this, I choose block style programming, specifically on the BBC Micro:bit.  The micro:bit was created for teaching introduction to computer science in Britain.  They goal was a microprocessor that had on-board physical inputs and outputs, the ability to connect to other components, and could by taught by middle school teachers, who had a wide range of comfort level in teaching technology, and likely no comfort in teaching programming. Blockly style programming (using makecode) is particularly suited as it had enough range to allow the application of computer science basic concepts, but without issues such as syntax, which in normal text based programming often dominate an introductory computer science course.  (in computer science, the concepts are the important part, syntax is merely the implementation.  The focus in time and energy on syntax is considered a distraction)

For materials, my mainstays of Adafruit and SparkFun were both unavailable.  Adafruit was designated an essential business, but was focusing on medical suppliers. SparkFun was closed. So we had to source kits from Amazon.com, who was working from stock.  The micro:bit itself was available, but kits tended to be from one of the multitude of unknown suppliers.

For curriculum, I used the book micro:bit in Wonderland from Tech will save us.  The target audience was people who could not program, and it was craft based, where the projects all had a programming component and a craft component (that interfaced with the micro:bit board through alligator clips and the touch pads. As an added bonus, there was a motivating reading, from Alice in Wonderland (which many people already owned, but was also free as it was in the public domain on Project Gutenberg.)

Some unit notes:

• Impacts of shutdown. The micro:bit was generally available on Amazon.com, but the kits of basic supplies were not.  There were two sources that were left, the https://www.techwillsaveus.com/site and various no-name kits on Amazon.com  The no-name kits tended to be breadboard based kits as opposed to alligator clip based kits.  For elementary and middle school classes, the alligator based kits would have been much better
• Block coding on micro:bit was something the kids took to easily. Even the one who had no programming experience (meaning not even having background in Scratch).  Using makecode was easy. By the second week everyone had a microbit (not necessarily a kit) and getting working makecode onto the micro:bit was not hard.
• First hard lesson was the control of the LED light, which was the first lesson that got off the micro:bit board. As this was really the first experience in physical computing, I believe that this is also considered a first milestone in electrical engineering.  It took a lot of one-on-one attention to get everyone through this.  But all of the other physical computing lessons (touch pads, connecting to sound speaker, other LED light projects) went much easier.
• If-else was not a problem
• Kids enjoyed the sound projects. Both the buzzer, then the music box.
• They also had no problem working through complicated if-else trees 
• The big assignment here was to use the microbit as a musical instrument, which meant making lots of sounds (kids like noise)

Really, the only difficult week was the first experience with lighting a light bulb.  While many students had experience with LEGO robotics and things like SPHERO, those kits deliberately hide the difficulties in connecting a controller to something off the controller. So after that, it becomes easier.

One thing I would do differently if I had to do this again, I would use the Sparkfun gator:circuit kit.  It breaks out many comment electrical components and packages them along with resistors (needed to make sure that low resistance components like LED lights do not get burnt out) to make a alligator clip friendly board, which is much easier than trying to connect components on a bread board (I think Adafruit has components that do this to, but not packaged in a named kit).  Leave the breadboard work for high school or later.

As this whole teaching kids programming experience was not nearly as traumatic as I thought it could be  :-)   there is going to be a follow-on.  But instead of having the focus be technology and computing, the goal will be to actually use these skills to do something, so we are going to take the Sparkfun gator:science kit, which has electronic sensors that can be used with the microbit to create data collection lab instruments, as the basis for an investigative science course. And in true co-op style, there will be a team of parents who will be trying to teach the kids science.  :-)

Data Manipulation with R by Spector: Book Review

Data Manipulation with R by Phil Spector
My rating: 4 of 5 stars

The quality that programming language based data analysis environments have that menu driven or batch environments do not is the ability to manipulate data. That means transforming data into usable forms, but it also means cleaning data, manipulating text, transforming data formats, and extracting data from free text. While R falls into this category of data analysis environment, almost all of the available material focuses on the application of statistical methods in R. This fills a much needed niche in how to process data. I still do not regard R as my goto tool for data manipulation, but this book means I am more likely to stay in R than otherwise. I used this as a textbook in a lower division data analysis course and the class went from a group that only half remembers Matlab to being able to process and analyze fairly large datasets. A comment I received was "I looked back on the work done in this project and I cannot believe I actually did that!"

The first part of the book is reading in data and writing out results. It discusses both text (csv, delimited, fixed) and working with relational database. One note is that the database they use is MySQL. This was easily convertible to SQLite, which is what I used in my class because my students are not IT savvy. I also used supplementary material for SQL (which is readily available) Then putting things together into data frames.

Next are a series of data types: datetimes, factors, numbers. For people who have only worked in Excel, these are deal breakers. Even using Excel, these are areas that often go unnoticed by students and lead to problems.

Character manipulation is about working with strings and a gentle introduction to regular expressions. For many of my students, they have never manipulated text programmaticly before, so this chapter was quite successful. For Regular expressions, well it provided a taste of it, enough to solve the lab assignment. I supplemented it with other material, but noone was going to learn regular expressions in 5 pages.

The best part of the book was the sections on aggregating and reshaping data. This is what made what my students were doing with R start to look like magic. Aggregations using the apply family of functions, reshape to convert data into long or wide formats, combining data frames, and an introduction to vectorization. This is not going to make anyone a functional programmer, but these are key idioms and Spector spent a lot of time here.

I am not going to prefer R over Python for working with text and manipulating data, but Data Manipulation with R shows how to do some non-obvious things. The examples are all interesting enough to be useful, and they all work as is. And this goes deep enough into some pretty powerful capabilities that expanded my students understanding of what is possible. While it is becoming dated (an update would have to include dplyr), the approaches it provides put the reader well on their way to being an accomplished R programmer, not just someone who feeds data into functions.

View all my reviews

Applied Predictive Modeling By Kuhn and Johnson: Book Review

Applied Predictive Modeling by Max Kuhn
My rating: 5 of 5 stars

I regard this as a more applied counterpart to more methodology oriented resources like Elements of Statistical Learning. So it applies machine learning methods that are found in readily available R libraries. In addition, the author is also the lead on the caret package in R, which provides a consistent interface between a large number of the common machine learning packages.

1. Built around case studies that are woven through the text. For each chapter, the math/stats is developed first, then the computational example is at the end, so that the example can develop data manipulation, application of method, then model evaluation. I like this as it allows for more complex and messy data sets than when using a new, small example for each problem. Also allows for better discussions when illustrating the differences between methods.
2. Data manipulation/data processing is given a separate chapter early on. I appreciate the attention given to working with the data (e.g. missing value imputation). There are other resources in data handling, but not in the same place as those that address the statistics methodology.
3. Emphasis on model evaluation. There is an early chapter devoted to model evaluation. Then each major section of the book has an early chapter devoted to model evaluation of that class of problem. This is in contrast to many books that are built around types of algorithms, and model evaluation is fit in. Methods and algorithms are relatively easy compared to the thought process of determining what is the right thing to do. It figures that this book will be strong in model evaluation when one of the authors is the lead on the caret package in R.

I used this as a supplement in teaching a data science course that I use a range of different resources because I need to cover working with data, model evaluation, and machine learning methods. The next time I teach this course, I will use only this book because it covers all of these aspects of the field.

View all my reviews

Testing in pairs: The Global Day of Coderetreat

Global day of Coderetreat: a day to celebrate passion and software craftsmanship

A Coderetreat is like a master class for computer programmers. It is a chance to view programming as a craft that can be practiced and honed.  And like other crafts, the way you develop skill and creativity is to create limits, then use your creativity to accomplish the goal while working around the limits.

As an engineer, I am not primarily a programmer, although I have some level of skill, so this is not the typical view of programming, which is viewed more as a tool and a necessary evil. The effect is that improved competency is not valued, which leads to  inability to deal with dirty data, models that cannot be implemented, and results that cannot be reproduced.

Code retreat is build around the four principles of simple design, due to Kent Beck: 

1. Runs all the tests
2. Expresses every idea that we need to express
3. Says everything once and only once
4. Has no superfluous parts

The structure of the Coderetreat is six sessions where we work with Conway's Game of Life. For each session, we pair with a different partner. In addition, in each session we are to begin from scratch, and there is a twist to the rules.  The goal was never to actually implement the Game of Life (although in two cases we actually had all of the parts working and tested), but to spend time working with someone else on code.

Observations

1.  Pair programming.  This was my favorite aspect of the Coderetreat, pairing with six different people.  I figure there were two where I was generally more skilled, two where I was generally less skilled, and two where we were pretty much even. In every case our end solutions had very different designs as it was a combination of our different ways of looking at things and our experiences of having tried different designs in previous sessions, and the skill levels of the people involved.  When I worked with students, we would occasionally have a session where we worked together to solve a problem, and some of my students have commented that they found those sessions to be invaluable because they had a chance to watch how I worked and saw how I dealt with different types of problems.  But this time I did pair programming on people on a much more even footing and I get to experience it as well.  It showed in how we used different tools (although I was experimenting with a new IDE), how we solved problems in code and how we solved logic problems. 

2.  Test driven development (TDD). I've heard of the concept before, and I have even contributed to a unit test framework, but I've never really done it.  What TDD did was to encourage more modular code. It also forced us to put more thought into our design, as we had to consider what information was required an in what format to do what we needed.  In one session, one member of the pair would write tests and the other would write the code, and the two were not allowed to communicate. As the one writing the tests, since we could not otherwise communicate, I realized that in writing the tests I was forcing a set of data structures and a design in my tests.

3.  Throwing away dsigns.  We started each session with a clean code base. What it meant was that we did each session using the lessons from what went before.  The first two sessions we did not get much progress, but the third was the one where we made the most progress, as we basically learned from the combined mistakes made over the first two sessions and designed the tests with the past problems faced in mind, and the solution was fairly easy after that.  That was good because the next three sessions were the ones with the wierder twists.

4.  New languages.  Python is by far my strongest language, but I did one session in Clojure and one session with Java.  In both cases I learned a lot about how people set up their tools and the idioms they used, which were different than what you see in standard texts.

5.  Dealing with constraints. There were three weird twists. One was mute pairs, one was limitations on the size of methods, one was no use of conditional statements.  Mute pairs forced the design to be simple and clear (especially difficult because we did it in Clojure, which I barely can say I know without the aid of a book in front of me), the size of methods led us to generate very ugly method/class hierarchy to deal with the extreme restrictions, no conditionals lead to a range of creative hacks.  This has an effect similar to a lot of exercises done in the creative arts, adding constraints is one way of encouraging more creativity.

This was a valuable experience. Most of the people there had computer science backgrounds, and pairing with them taught me a lot.  And I was somewhat glad to know I could add to people's knowledge base as well.

Thanks to Code & Supply (@codeandsupply) and Think Through Math (@ThinkThroughMath) for making this event possible and for helping to keep it free, and to IBM for hosting.

Flask Web Development by Grinberg: Book Review

Flask Web Development: Developing Web Applications with Python by Miguel Grinberg
My rating: 4 of 5 stars

I'm not a web developer, but Flask has always had an appeal of potentially being a potential front end to a database based application. But I've never gotten very far as tutorials generally look at only the main application, and I'm pretty sure I need some extensions but I have not been able to assess the quality of the many plugins available. Flask Web Development is that tutorial that shows Flask and selects quality extensions to introduce.

Flask Web Development is written as a tutorial, not a reference. As such Grinberg can decide on what is important. It starts like many other Flask tutorials in looking at the basic application structure, templates and web forms. But then it looks at databases along with a set of extensions for database management built around SQLAlchemy: Flask-SQLAlchemy, Flask-migrate. (and email, which I don't do)

He brings it all together with walking you through a blogging application. What gives the tutorial an over-the-shoulder feel to it is an innovative use of the github repository that goes with the book. Instead of having source files in the repository, the repository uses tags to incrementally build source files, so it is really like working alongside someone who knows what they are doing as they build the application. One tag will have a basic working version of functionality, and checking out subsequent tags builds out more features.

Flask Web Development covers many aspects of web programming, well beyond what most Flask tutorials will cover. I appreciate the deep dives into database management with SQLAlchemy, and the sections on testing and profiling which have applications beyond web development.

Disclaimer: I received a free electronic copy of Flask Web Development as part of the Oreilly Blogger programming.



View all my reviews

Clojure for Machine Learning by Wali: Book review

Clojure for Machine Learning by Akhil Wali
My rating: 3 of 5 stars

For a book that is about [programming language] for [computational task], there are two approaches. One is to assume basic competence in the language and teach the task, the other is to assume that the reader has basic competence in the task and teach how to do it in a language. This book assumes knowledge of Clojure and tries to teach machine learning. But I find that it does just enough to be dangerous. It has a series of methods but does not provide discussion on why you would choose once class of method over another, and it completely skips model evaluation. What this creates is someone who has a good idea of the mathematics and implementation of methods, but not when to use it or if it actually did what was intended.

This makes me slightly different than the ideal audience of this book. I am learning Clojure and have only started using it for data analysis in real life. However, I have used Python and R for data analysis for several years now, and I have use both of them (and trained students to help me using both) for different machine learning projects (and I use R for teaching a course in data science).

Each chapter in Clojure for Machine Learning takes a look of a class of machine learning algorithms and takes several looks at it. Generally, Wali looks at the mathematics and theory of the algorithm, then a simple implementation in Clojure, then some examples of using existing library implementations on a problem. The mathematical treatment seems nice, but it would not compare to an actual text on machine learning/data mining. And while seeing an implementation in Clojure has some value, I would have liked to have seen more humility in doing so (i.e. some recognition that there are limitations of an implementation that can actually fit into a book of this size with everything else that needs to be done).

Two things that bother me about this after I finished was realizing that for each class of algorithms, the book only covers one or two methods. Which is fine, but it does not even acknowledge that there is a greater world. And as there is no discussion on how to perform model evaluation, an enthusiastic reader may reach the conclusion that they know what they are doing when implementing them against a data set and problem. Essentially, the enthusiastic reader knows enough to be dangerous and does not know what he does not know. If I were to suggest this book to someone, it would have to go with a severe caveat that what you know after this is how to set up a machine learning problem. More research has to be done to determine what actually needs to be done (the libraries used are much broader than what is covered) and then, learn from somewhere else how to evaluate or tune the methods used.

In the end, I would treat this as a book of examples or cases of Clojure being used in machine learning. There is room in the world for a book on Clojure for machine learning, but this is not it.


View all my reviews

Java Cookbook by Ian Darwin: Book review

Java Cookbook by Ian F. Darwin
My rating: 4 of 5 stars

What you want from a programming language cookbook is instruction on the basic tasks that are needed to form the scaffolding that you build around your application. Language teaching and references can teach you syntax and good practices. Topical books can demonstrate how to tasks in the large. But the cookbook is for the small but necessary tasks. And for me, who does not spend much time in the JavaVM ecosystem, the Java Cookbook is a very welcome addition to my bookshelf.

I spend most of my time doing scientific programming in Python and R, but I am starting to return to the JVM because of the need to deploy what I develop. But while other languages like Jython, Groovy, Scala, and Clojure exist on the JVM, to use them well means you need grounding in Java, certainly most of the instructional material assumes more than passing understanding of the JVM and the Java standard library.

I've been building a prototype application using Java as my means of re-learning Java. Where this cookbook has helped me already is in understanding better how to configure projects, more effective use of the Java data structures and I/O, and some utilities. While I know what I need to do through my experiences in other languages, and Java tutorials and references can identify the libraries and functions that I need, the Java Cookbook provides well written examples that I can use to guide me through the JVM.

There are some warts. This book (like most JVM books) seems to be written with the understanding that the readers are web programmers and I think that the discussion of the options available are filtered with that in mind. But this is a very good reference for those times when you know what you need to do, and it is not the type of thing that gets put into a tutorial.

Disclaimer: I received a free electronic copy of this book through the OReilly Blogger program

View all my Goodreads reviews

High Performance Python By Micha Gorelick, Ian Ozsvald: Book review

For someone like me who is a technical programmer but did not study CS, I've seen hints on how to speed up Python numerical code, but I only had a vague understanding of the principles and application.  This Early Release version of High Performance Python has examples that demonstrate why certain data structures are faster than others in particular situations, and how to use the various data structures provided.

But what may be unique is the chapter on the ways of speeding up Python through compiled code. There are many ways of using compiled code through Python, C and FORTRAN extensions, Cython and PyPy, and more recently Numba. But this book explains the strengths and limitations for each, along with a number of other ways of using compiled code along with Python that I had not heard of before. There are many references for each of these, but no general overview of this group of resources.

This is still an Early Release stage, so there are some warts. Many of the code examples are raw, and you have to know what you are doing to fill them in and get them to work. A website or Git repository with the source code examples would be very helpful.

Disclaimer: I received a free electronic copy of the Early Release edition of this book as part of the O'Reilly Press Blogger Program.

Notes from teaching data science for the first time

Drew Conway Data Science Venn Diagam

I spent this past semester teaching a course in data science. While there has been a data mining course taught in the department, it is offered irregularly and had a different focus.  The premise for the course I taught was that data science was the intersection of data hacking, mathematical and statistical methods, and domain knowledge (with props to Drew Conway). The students I had generally had little to no programming experience (or meaningless background). All have had a first course in statistics.

I used two texts. First was Stanton Introduction to Data Science, which is used in the Syracuse Data Science certificate program.  Second was Introduction to Data Mining with R by Luis Turgo.  All of the students were also told to go through Introduction to R prior to the beginning of the course (or as early as possible).

The class started off going through Introduction to Data Science, which included a few introductory chapters to data analysis, and introduction to R and the R Studio IDE.  Then were chapters on some basic methods at the basic level such as text processing, review of regression.  Then additional methods such as association rules and support vector machines.  We then switched to Data Mining with R which were a series of case studies.  Each case study had some form of data munging (manipulation) required, with the first one having an involved demonstration of how to handle missing values, either determining the correct value or removing as appropriate.  Each case also had a lengthy discussion of the methodologies used, with what each is being used for and a basic understanding of how it worked and its implementation using libraries in use with R (there is a book package, but it has mostly data sets and some functions to assist in data manipulation and visualization.

The assignments were built around individual projects. Their were three presentations, exploratory data analysis, preliminary data analysis, then final. The first two they could work together, but the final one had to be solo as they needed to have individual topics (even if they used the same data sets). The intent was that these assignments would build towards a final goal (but they had flexibility to bail if they wanted to mid-semester.

Probably 1/4 of the students found projects off of Kaggle, which is useful because it has a nice complex data set and comes with a legitimate question.  Another 1/4 of the students used public health as a motivating area (University of Pittsburgh is home to Project Tycho, which is a rich dataset of infectious disease in the U.S., also, there is a joint program with the Department of Industrial Engineering and the School of Public Health).

Some problems that came up. First, I discovered that many of the students had an operating assumption that all data was normally distributed, and they constantly made claims that their data was normal. Even when the data was noticeably skewed.  This was embarrassing when they would make statistical tests, and the test graphic would include the corresponding normal approximation which was nowhere near the data. I eventually figured out that for many of them, when they took statistics they were constantly fitting normal distributions in their homework data sets, so I explained that their textbook problems were written so that their would be a normal distribution to find.

Another problem was the lack of a hypothesis.  Many students started to pick problems that could be solved through linear regression and declared that because it met a p-value criteria they were done. (and in some cases, I recognized the data set as being a teaching data set). But even though they could fit a regression, there was no theory on why the data related in a given way. Essentially, they were pushing data through an algorithm without any subject understanding.  Most (not all) of them got the ideas by the end of the second presentation.

A third difficulty was skipping the model evaluation.  Most of the methods covered have some parameter that was the analysts choice, so they should have explained how they chose the value of that parameter.  Generally, this should have been a discussion of making the tradeoff between closely approximating the observed data and overfitting.  Some students skipped this completely (essentially, this is what would happen if you fed data to an algorithm then reported the result using all default values)

One big observation I had by the first presentation was being able to identify the level of programming ability by the choice of projects.  I strongly suspect that a number of students were minimizing the programming required.  But that became reflected in the level of ambition of the projects.  Non-programmers tended to choose simplistic data sets with little variety.  I think the difference is the workload.  People who could program were able to slice the data available on a multitude of dimensions without regard to scale, since the computer would do all the repetitious work, while those who could not program generally were reluctant to have large sample populations or multiple data sets on the same population.

Things for next time.  First, impress on them the need to learn to program.  Essentially, the projects from those who could program were so much richer than those who could not (even at a low level of programming skill) that I was embarrassed for those who could not program.  Second, I should push harder on the need to have a hypothesis that was driven by domain understanding of the problem. This should be pushed harder from the very beginning to discourage people from merely pushing data through statistical methods and reporting results.

For teaching data mining, I think that the organization of the course needs more methods focus. The principle text was case driven, but that meant that methods were being introduced in a fairly arbitrary sequence.  I ended up doing a methodology focused review over the last few weeks. What I should do next times is after the introductory section (Stanton Introduction to Data Science), have the next several lectures be a tour of the classes of data mining methods (regression, classification, clustering, feature selection), then do the case studies.  One resource I found useful in this are articles from the Journal of Statistical Software, many of which are focused on R packages that implement classes of methods.

This was a very good course. I wished that the students did more participation (by the final presentation, there were some points that were given based on shear quantity of comments, which several students took advantage of). Some of the projects were much more ambitious than any other done in the MS program. And I have a lot stronger argument about the need for the graduate students to know scientific programming as a skill set.

Data Mining with R by Luis Togo: Book review

Data Mining With R: Learning By Case Studies by Luís Torgo
My rating: 3 of 5 stars

Data Mining With R (DMwR) promotes itself as a book hat introduces readers to R as a tool for data mining. It teaches this through a set of five case studies, where each starts with data munging/manipulation, then introduces several data mining methods to apply to the problem, and a section on model evaluation and selection. It fills a place in the literature since it devotes a lot of space for data manipulation before applying the various methods and model evaluation afterwards. But it is hard for people learning data mining since it spreads the types of model throughout the book.

I used this as one of two texts to teach data science to people whose programming and data analysis skills were generally at a very low level. The big advantage of using a programming environment such as R for data mining is the fact that you can do data manipulation in the language, then apply the methods. Many of my students have taken machine learning elsewhere, but they always used prepared data sets, so this emphasis on data manipulation with several very disparate data sets is a unique feature.

The second big advantage of this book is the focus on model selection. For each chapter, the book goes through the exercise of determining which model should be used, and how to diagnose the model to determine which one is appropriate and best for the problem. I especially appreciate the fact that in some cases, the conclusion of the book after model evaluation is that the method did not work for the problem and question at hand. Because most textbooks focus on demonstrating that you did find something, in some cases my students get confused when in real problems they did not find an effect.

Where the book is lacking is the fact that the methods are scattered across the case studies with minimal organization. While this is a result of the realities of the cases, the book would have benefited from a roadmap chapter or introduction that gave methodological context (i.e. what methodologies are used in the book and where they are). This lack made it very difficult to use as a textbook, and by the time I was done using it I was essentially building the roadmap to use the book. This makes it not useful as a standalone textbook for such a course, but very good if there is another text that gives the overview of the methodologies.

View all my reviews

Agile Data Science by Russell Jurney: Book review

Agile Data Science: Building Data Analytics Applications with Hadoop by Russell Jurney
My rating: 4 of 5 stars

One of the problems with data science is that any description of what is encountered takes on the appearance of a mythical unicorn, noone person could possibly have all of the skills required. And it gets worse when you add to the standard set of statistics, domain knowledge, and programming the ability to deploy the application into a high speed environment. This book is not going to make a data scientist an expert in running a data center, but it is useful to give someone who has the rest of the skills an understanding of the environment their work will be deployed into.

One of the conflicts between the data scientist/analyst and information technology groups is that while the data scientist gives the data owned by the organization its value, IT is charged with storing the data and providing the access. And in a high velocity, high volume environment of big data, not understanding how the architecture works can lead to the data scientist creating valid solutions that cannot be applied in the actual day to day working environment. That is where this book comes in. The book has associated virtual machines in software repository so that the data scientist who does not know anything about infrastructure and the software stack that the data and the analysis rides on can see how everything fits together.

The book title is misleading. This is not a book about data analytics. This is a book for data analysts so they know how their analytical application is deployed and applied to day-to-day use in enterprise environments. For that reason it is useful.

View all my reviews
Disclaimer: I received a free electronic copy of Agile Data Science as part of the Oreilly Press Blogger program.

Mining the Social Web, 2nd ed by Matthew Russell: Book review

Mining the Social Web: Data Mining Facebook, Twitter, LinkedIn, Google+, GitHub, and More by Matthew A. Russell
My rating: 4 of 5 stars

The hardest part of learning a data analysis method is not in implementing the method, it is applying the method in the context of a real data problem. And data mining and machine learning texts often skirt the issue by using pre-processed data sets and problems defined to fit the method being taught. Russell uses analysis of social media sites to set a context where you start from having to gain access to real data sets, clean and transform the data into forms that your analytical libraries can make sense of, and then use the results to make a conclusion. For that, it rates a place along any other text that focuses more on the analytical methodology itself.

What I most appreciated about this book was the work put into converting data from one format to another. From the beginning, when he works with data pulled using a services API, then getting that into a format that another library requires, then getting those results into a data mining framework for analysis. Following his flow has helped me understand the methods better. And these examples of processing data from format to format is something that gets my students stuck before they get really started in a project. I especially appreciated the chapters that worked with the Natural Language Toolkit (NLTK) and the NetworkX graph libraries. These examples helped me get pass what was the hard part for me in working with these libraries in previous encounters.

The virtual machine is also very helpful. I have always found the hardest part of working with Python for analytic computing has been teaching my collaborators how to get set up. And in data mining this is even harder than standard. I was able to get through his book installing everything on one machine, but on another I used the author's virtual machine, and I have pointed a student who was working with me to the virtual machine as well.

This is a great book to work through the mess of implementing data mining methods in real situations. It is not a theory book, but it serves its purpose well.

Note: I received a free electronic copy of this book from the O'Reilly Press Blogger program.

View all my reviews

Doing Data Science by O'Neil and Schutt: Book Review

Doing Data Science: Straight Talk from the Frontline by Rachel Schutt
My rating: 4 of 5 stars

Doing Data Science is about the practice of data science, not its implementation. It is based on a course on data science that featured a guest lecturer on each topic. This leads to the guest lecturers (and chapters) focusing more on important concepts rather then the methodology. So, this is not a textbook or a how-to-do-this type of book, rather it is a how-to-think-when-doing book.

A problem with books like this where each chapter is written by someone different is the need for coherence. A second is that each author typically has something to day, and she has to say it in her chapter. So, compared to other data science books, it suffers from the chapters not building on each other in a systematic way and having multiple messages that appear as you go through the book.

One benefit from this is that each author has something to say. While I find the book thin on how to do things, this is a good source of wisdom in why things are done and issues that come up along the way in real life. I am teaching data science for the first time and I find myself turning here for topics of discussion which my chosen textbooks don't cover (as they have more focus on how to do things).

I don't think this is the book to use to learn how to do data science, and I suspect the students at Columbia learned how to find other sources to help them figure things out. But it provides wisdom, which is harder to find and worth quite a bit.

Note: I received a free electronic copy of this book from the publisher as par of the OReilly Bloggers program.



View all my reviews

An Introduction to Data Science by Stanton: Book Review

An Introduction to Data Science by Jeffrey M. Stanton
My rating: 4 of 5 stars

This freely available book fills a nice little niche, people getting started in data analytics. The first problem people have in learning this is they tend to learn a number of techniques, but in practice they cannot get past the step of accessing and preparing the data. This book covers this and gives good practice in it. I plan on using this as the first of two texts for the course. This book will get them started (gently) in R and accessing and processing data, then a case based data mining book where they can use what they learn here to work with larger data sets that are then analyzed in the methods that the other book goes into more depth.

This book would be for those who are just getting started and need some hand holding as they get started with the R environment as well as those who don't really know where to get started with new data sources. It is most useful for those who are starting from nearly scratch, but also for those whose education included data analysis techniques, but whose training neglected those crucial steps on how to get started and why you should be using some class of method, not just the how. And since it is a free electronic book, it is a low cost way to get started.

View all my reviews

Learning Java 4th ed by Niemeyer: Book review

Learning Java by Patrick Niemeyer
My rating: 3 of 5 stars

I remember using an earlier edition of this book to learn Java many years ago. I even used Java for writing simulations as part of my thesis. Since then, since I generally do scientific and technical computing, I have generally used Python and R linked with C, C++, and Fortran libraries instead of Java. But I have used languages on the Java Virtual Machine (Jython and Clojure) and I probably need a refresher on how the JVM works. This book does give a good overview of all of the scaffolding that goes along with programming in Java, but its focus on the language proper and web and GUI programming in particular leaves me wondering how to get real things done with it.

The first part of the book is what seems to be the standard first chapter of almost all programming books nowadays, a argument of why to use Java. While some things like the safety aspects of the JVM ring true, when he talks about the various dynamic languages he gets some basic facts wrong and mis-characterizes how these languages are used in practice. It would have been better if he did not include these sections at all because he frankly started loosing creditability here. (coming from Python, the general line is that Python is at its best as part of a two language solution, and Java is on the list of likely languages to pair Python with along with C, C++, and Fortran.)

Next are several chapters on setting up your machine to develop and run Java. And several chapters are required. I found many things that I often have to spend several hours looking up whenever I start or deploy a JVM based project so I'm glad that someone realized that this really needs to be in the beginning of an introductory Java book. Things like IDE's, setting up classpath and other environmental variables, and the whole java toolchain. While I applaud that this is required, it somewhat gives a lie to the idea that Java is a simple write one run everywhere tool.

The rest is a tour of the Java language. Data types, statements and expressions, exceptions, assertions, classes and objects. I was specially interested in the discussions on Generics and Threads, as I had not used them before. It could be that I'm spoiled by how Python handles the equivalent of Generics, and both Python and R multi-core libraries, but this seemed very detailed and complex (not helped by Java seeming to require that everything be declared in duplicate).

Despite the title, this felt like more of a reference than a tutorial. Having several chapters on setting up the scaffolding that is needed for every Java project is something that some places seem to gloss over as easy (especially if the IDE does it for you), but makes this book a handy reference. But in the advertise role of learning how to use Java, I'm still skeptical.

Disclaimer: I received a free electronic copy of this book through the OReilly Blogger program.

View all my reviews

Python Cookbook, 3rd edition by Beazley and Jones - Book review

Python Cookbook by David Beazley
My rating: 5 of 5 stars

The third edition of Python Cookbook is part of what seems to be a general trend of issuing new editions focusing on Python 3. For me, whose focus is on data analysis and technical computing, this is the time to be thinking about the change from Python 2 to Python 3 as the base libraries of Numpy, Scipy, Matplotlib, and iPython have been ported, and the various other libraries I use that depend on these are being ported as well. But this edition is not just porting the old cookbook, it is a complete rewrite to go with the big shift that was Python 2 to 3. Because a lot of what I used the old cookbook and many of the recipes at the ActiveState website was for handling issues related to crossing versions (I had some projects with Jython, which was several versions behind CPython) and ways of getting around issues that are purportedly solved in Python 3. For that the third edition of Python Cookbook fills its purpose of showing idiomatic ways of performing some programming tasks, and being a reference for how to do thing well and elegantly taking advantage of the language and libraries, not fighting it.

I focus on using Python as a data analysis platform. So I generally only learned as much of the language I needed to in order to use the scientific stack of Numpy, Scipy, Matplotlib, Pandas, and the libraries that were built around them. But that means that I have not gotten to know large portions of the standard library. And introductory books don't cover this either, they focus on using the language itself. There are also a large number of books and references that focus on Python as a web development tool or a system administration tool, so those parts of the standard library get a lot of coverage in teaching materials. But the rest you almost have to stumble upon. In ideal conditions, the way you would learn about much of the standard library is to have someone who was more experienced nearby show you what you needed to know, as she demonstrated methods in her code that did the things that you never learned in class. But sometimes there is no such person. The Python Cookbook plays this role, of demonstrating how to do things in Python 3 that are practical and you probably would not learn while learning the language itself.

Some areas that I found useful are the heapq, generators, and the I/O. heapq is a module and data structure I just never got around to learning. Usually discussions about Python data structures made their way to deque and heapq was discussed by reference. But after looking at the priority queue discussion, I fired up an iPython notebook and went through every recipe that used heapq and I've started thinking about how to rewrite a model I recently coded up. Generators and File I/O are areas that I knew in passing through my use with them in data analysis, but Python Cookbook opened up new ways of understanding (I am starting to get why JSON is so useful). Now, there is nothing special about these, but seeing these parts of the standard library in use in an elegant way is something beyond what you would get from a module documentation or a standard tutorial.

I do miss the introductions to each chapter that was in the 1st and 2nd editions of Python Cookbook. But what Python Cookbook does give is an idiomatic feel of using Python3, when there are not all that many mentors out there to go around. So this is something very useful for others who are starting to use Python 3. It is not for learning the language, but it is for using the language well.

Disclaimer: I received a free electronic copy of this book as part of the Oreilly Bloggers program

View all my reviews

Interactive Data Visualization for the Web by Scott Murray: Book review

Interactive Data Visualization for the Web by Scott Murray
My rating: 4 of 5 stars

This is really a d3.js (Data Driven Documents) book, since the entire book is about using the d3 Javascript library for data visualization. For me, coming in with a background in data analysis but no practical Javascript background, this provided a good overview of many Javascript/web concepts required then the d3 library.

My background in data visualization is R (ggplot and lattice) and Python (Matplotlib). I find Excel too limiting and Matlab inflexible. But what I'm used to are descriptions of visualizations that are then implemented. d3 is different, it makes you specify the form of the visualization and makes you work at a very low level (which is what ggplot and Matplotlib are trying to avoid). But this provides room for great creativity, and learning how to do this is the point of this book.

This book does a good job of introducing enough of the background technologies of Javascript and the web to make d3.js useful. This was always the reason I never got around to actually using Javascript, I never learned all the things you needed to know to make it useful. For example, I now think I understand JSON well enough that I can see using it as a data store for data frames instead of CSV or SQLite or pickles (Python).

I come at this as someone who already does data analysis, but may need more ways of delivering the products of the analysis to others. And this has promise as a means of this delivery through a browser, which is already comfortable for most. And this book does a good job of working both the subject of d3.js and the web ecosystem that it is a part of to get to someplace useful.

Disclaimer: I received a free electronic copy of this book as part of the O'Reilly Blogger Program

View all my reviews at Goodreads.

A recap on work flow, note takers and task organizers

Some experiments I have been doing with workflow over the past semester as introduced on this post back in January. 

1.  Using Pandoc, Markdown, and pweave for class preparation.  

I did the notes for my Logistics and Supply Chain course completely in Markdown.  This included slides for in class presentation, writing homework solutions, and everything else with the class preparation.  I wrote a Makefile that took care of everything.  Basically what it meant is that to create slides for class prep I wrote it as a text file.  It does force some simplicity because there is no ability to tweak a slide or an image.  But that is probably a good thing because it forces efficiency.

One nice advantage of using pweave for homework and exam solutions is that it makes it easy to have multiple versions of an exam.  If I have equations and steps in place, I can change a few parameters, and maybe an assumption or two and I have a new problem with not much extra work.  Also I did a lot of cut and pasting of code.  The code I wrote for presenting in class became the code that I used to solve the homework.  And since I let them see the code (and I also copied over the equations needed to solve the homework) it connected the homework solution to the lectures.  One issue is that so few of my students were competent programmers in any language that the Python code was not all that helpful to them (other than proving that I did indeed solve the problem)  But this was a definite success.

2.  Using Mercurial as a hosted version control system.  

I had to work with a new team on one project I was working on, so the first thing I did was to move the  former post-doc's work into a version control (Mercurial) and put it on a hosted repository (bitbucket).  And we spent a month refactoring the entire thing to separate out the code from the site specific bits (because we were going to apply it to a new location).  The repository meant that we were able to be aggressive in refactoring (yes, I did have to completely through out a day of work at one point because what I was thinking was not going to work.  All I did was wipe my directory and reclone the repository)  The others got into the hang of putting everything in the repository as well.  Now, the comments are not all that good on the part of the post-doc and grad student working with me, and we tended to work in separate areas so it was not true collaborative programming, but it was not bad.

I plan on continuing this over the summer with a few students who are learning agent based modeling together.  The intent is that if all of us are putting our models in the repository, we can ask questions like "why does ___ not work" and we can have their model to look at.  We'll see how that works.

3.  Google Tasks as a task organizer  

This worked out pretty well.  I ended up using Google tasks as a note taker (more on that later)  I was off an on regarding being consistent, but there were periods where I had a lot going on that this to-do list being available on all of my computers, my iphone and ipad was very useful.  Keep doing this.

4.  Tomboy as a notes organizer. 

Note taking sounds like a no-brainer.  Not.  This one pretty much died.  In February Canonical eliminated an API that enabled use of Ubuntu One as a storage place.  So I ended up storing the notes data on Dropbox (so my Windows and Linux machines were synced).  But the iPad app that integrated with Tomboy no longer worked, so if I was using the iPad, the notes were not integrated.  And this pretty much meant I was not using Tomboy notes.  Sometimes I would create a task in Google tasks and create a note if I needed something.  But generally this went unused.

I'm going to try this again.  But since I realize now that having it on my iPad and syncing to my computers is the key to the whole affair, my main focus was on the iPad integration.  So I am looking at Simplenote for the note-taking. It is originally designed for iPhone/iPad, and there are applications on Windows and Linux that integrate with it (and also the Google Chrome browser)

Fernando Perez: "Literate computing" and computational reproducibi...

Fernando Perez: "Literate computing" and computational reproducibi...: As "software eats the world" and we become awash in the flood of quantitative information denoted by the "Big Data" b...


I am teaching a course in logistics and supply chain.  Because the course is focused on the modeling of supply chains, and the students are generally using Excel (I actually did all of the homework solutions in Python) I used the Reinhart and Rogoff’s “Growth in a Time of Debt” discussion as a mini-case (every class I find an article to discuss.  Usually it focuses on business decisions, but this time I chose this topic.)

The focus on the class discussion was on how you look for errors.  Because Excel makes this nearly impossible, the real question was on how you focus your time.

But the real answer, as Perez mentions, it to use tools that make reproducible research simple.  The homework assignments I gave resulted in spreadsheets that covered 4 tabs and were a couple hundred columns and thousands of rows.  My solution in python was about a page of commented code which had a near one to one correspondence with the mathematical formulation of the model (with the addition of a few lines to read and massage the data.)  The spreadsheets were pretty much un-auditable.  My code had comments that would tell a reviewer what to look for.  Even for smaller problems, my students spreadsheets were fairly obtuse while my Python programs (I use Pweave) alternated between the explanation of each step with the calculations described.

The Thomas Herndon, Michael Ash, and Robert Pollin paper included R code that both demonstrated the effects of the errors of Reinhart and Rogoff.

http://www.peri.umass.edu/236/hash/31e2ff374b6377b2ddec04deaa6388b1/publication/566/

For fun, Vincent Arel-Bundock has converted the Herndon, Ash, and Pollin into an iPython notebook, suitable for looking at (and reading) as well as downloading it and playing with the data to test any thoughts someone may have about testing the impact of various types of errors.

http://nbviewer.ipython.org/urls/raw.github.com/vincentarelbundock/Reinhart-Rogoff/master/reinhart-rogoff.ipynb

Review of the R Graphics Cookbook by Winston Chang

R Graphics Cookbook by Winston Chang
My rating: 4 of 5 stars

If using the grammer of graphics as implemented in ggplot2 is like learning a new language, the R Graphics Cookbook is not a book that tries to teach you a new language, rather it is like learning a language through using it and is a different take on ggplot2 and graphics in R than other ggplot2 books.

ggplot2 has always presented itself as learning another language. And while it seems that a grammer of graphics is the right way to go I have always had a hard time getting a handle on it. But while the idea that you can build graphs through a grammer with a consistent meaning is elegant, sometimes you need to start by accomplishing a task. R Graphics Cookbook becomes very much like a phrase book you need to get started. Some of the earlier chapters cover categories of graphs and work you through the variations. Other chapters focus on the graph annotations, titles, axis, labels, etc. And since this is a grammer, you are assured that this is applicable to all of the types of graphs that were covered earlier.

Another aspect of this book that is helpful is the chapter on data munging. While this book focuses on graphics, the principle library, ggplot2, requires that data has been shaped into data frames before using it. But this becomes an overhead that I'm not used to coming from other graphics and plotting paradigms such as in Matlab, Python, Excel, etc. So the chapter on getting data into shape is important. This includes creating data frames, creating new data frames for purposes of generating graphics, and modifying data frames so that they yield more elegant graphics.

I still think I will have to understand the gglot2 book to understand the grammer of graphics in detail, but this book is better for me to get work done, and may make the difference between using ggplot with its elegance rather than other graphics libraries that I use when I get frustrated by the overhead to get started.

View all my reviews