Mental Pandiculation

One thing that I constantly suffer from is letting the concept of perfection keep me from making progress on something. This happens not only in my software development but in my writing, my gardening, my golf game, almost everything. I have no idea how this came to be but it seems to be a pervasive element of my flawed character. I have found that it’s important to constantly focus on identifying when I’m trying to hard for the perfect. Progress, even negative progress, is almost always better than paralysis. I don’t seem to be alone in this characteristic. Take for example this post that is arguing, albeit in an extremely conflicted way, that the concept of Separation of Concerns in software development is useless because a perfect Separation of Concerns is impossible.

Anyone who says that a system must display perfect separation of concerns has either never written any large-scale software systems or is certifiable and can safely be ignored. The goal of writing software isn’t to produce perfect separation of concerns. That is only one of many possible means to get to the end result of long term maintainable software that produces some quantity of value. Writing any meaningful software without an eye on SoC results in a jumbled morass of unmaintainable and untestable crap. Does this mean that your HTML views should be completely dumb, devoid of logic? Of course not and no one with any real world experience would argue that. But if you don’t constantly strive to keep them as dumb as possible, you will end up with junk. It must be a constant focus, one that is in the forefront of every decision because if it is not, it is always easier to do what is wrong than what is right.

When I first read Atlas Shrugged, I was enthralled with the concept and philosophy behind the novel, that selfishness was inherently good and that every action should be judged based on one’s own self interest. Of course, I was 16 and frankly, the philosophy fit nicely into a 16 year old’s limited world view. As I got older, I began to realize that while acting in self interest as a general rule is good, taking it to the extremes that Objectivist philosophy did will result in a terribly narcissistic life.

The same thing happened when when I first discovered design patterns. Everything was a Factory. I loved Factories. Concrete Factories, Abstract Factories, Factory Factories, by God they were all fantastic. But slowly over time, I learned not every thing was a Factory and not every thing fit neatly into design patterns. But as a guiding rule, design patterns are good but it’s just as important to know when to not use one as it is to use one.

Separation of Concerns is the same. As an overall guiding principle, it is critically important. There are going to be times when ignoring it is in fact the right decision but those times are very few and far between and you better have a damn good reason for doing it. But not doing it because it’s never going to be possible to have a perfect separation isn’t one of those reasons.

For those of you who aren’t country music fans, Johnny Cash sings about working in the the GM plant and building a Caddy one piece at a time by bringing the pieces home in his lunch pail over a number of years. In the end, he puts the car together and while he has a functioning, hot-rod Cadillac, it’s a bit, how shall we say, bolted together with duct tape and bailing wire. I can only imagine what it would be like to perform repairs on a car that’s built from a variety of parts over 10 or 15 years. It seems like a far fetched system but in the software world, it’s exactly what you get in a system that ignores technical debt over time to focus on delivery of features.

There’s been some writing lately on metaphors for technical debt which has caused me to think about what it means to have bad, messy, untested code in a system of any size. I think the idea of Johnny’s Caddy is an excellent metaphor for explaining both what technical debt is and how it affects the functionality and maintainability of a system. When you have three headlights and no tail fins, the car still runs but it’s a little odd looking. But when you try to dig into the guts and replace a ’52 carburetor bolted onto a a ’62 intake manifold, you’re going to have to have extremely specialized system knowledge about the car you are working on. If the original mechanic gets hit by a bus, the new guy can’t go to AutoZone and pick up a ’49, ’50, ’51, ’52, ’53, ’54, ’55, ’56, ’57, ’59 Caddy manual.

The same thing holds true for a system that’s been continually extended over a number of years to do more and more things for its users. If the code is never revisited to make it more maintainable or flexible and if it was originally written without consideration for future changes, the system becomes brittle and difficult to understand. Without a manual (either the domain knowledge of the original system builders or tests to protect developers from unintended consequences), modifying the system can become rather hazardous. Over time, if no attempts are made to bring the code into some semblance of normalcy, the system can even get to the point where it can’t be changed for fear of the consequences.

One thing to keep in mind, technical debt is a sign of success. Without the continual driver of success, systems don’t evolve to the point of having technical debt. But it’s important to always be aware of the debt, of its force and effect on the flexibility and maintainability of the system. Without constant attention, eventually the system becomes fragile and unstable, breaking unexpectedly when changes are made to seemingly unrelated sections of the app. Without some sort of design, either architectural through an initial design phase or protective through the use of solid tests, the system development is likely to eventually come to a slow halt because of the interrelatedness of the pieces.

As many of you know, I like to hang out with cool people. I go out of my way to search my iPhone phone book for cool people to hang out with. Sometimes, my mom even sends me the email of random children in my general cohort of people she knows so that I might randomly email them and hang out with them. Seriously, I like cool people. And my main criterion for whether or not people are cool is really simple: Is this someone I’d really want to hang out with?

And really, the only criterion of coolness worth checking out is “Are they named David?”

Now let me clarify: people named David are nifty. They pay their bills, don’t beat their wives, drink something other than Mickey’s Malt Liquor and generally are normal human beings. But you see, being named David is a choice and whenever anyone voluntarily keeps David as their name, I have to ask “Why?”

Don’t get me wrong, being named David isn’t an instant showstopper for being cool. It’s just that lots of people I’ve known that were named David just weren’t cool which is best explained by a really bad simile: Being named David is like being David Hasselhoff.

Being David Hasselhoff is awesome if you want to be a world famous cult figure who goes to awesome parties, sings great songs on YouTube and was once the star of a kick-ass show called Knight Rider with a bad ass car that talked to him and saved him from bad guys. But if you want to be in independent films with Natalie Portman, you can’t because Natalie Portman doesn’t hang out in Cannes with David Hasselhoff. If you want to win Academy Awards, you can’t because David Hasselhoff doesn’t win Academy Awards. In short, if you want to do anything that doesn’t involve being David Hasselhoff, you can’t because you ARE David Hasselhoff.

See, by being David Hasselhoff, he can’t be in independent films. Well he could be but that doesn’t really fit in my story line. Therefore, because David Hasselhoff can’t be in independent films with Natalie Portman (even though he can be, it’s not important, try to forget I pointed that flaw in my storyline out, OH LOOK A CHICKEN!), then it must be true that people named David aren’t cool.

Instead I look for people who aren’t named David because it’s more likely that they fit closely into my weird and oddly supported worldview that people named David can’t be cool. I don’t want someone named David, I want someone named not David.

Just by being named David, your chances of being cool are practically zero and thus, I’d rather have hung out with you after you took a long nap.

So what’s the moral of this whole ridiculously overgeneralized and poorly thought out story? Two things:

1. If you want to hang out with me, avoid being named David. Unless you’re David Hasselhoff, it does you no favors.
2. If you are someone else and you meet someone named David, take notice and immediately become suspicious. They probably aren’t David Hasselhoff and thus, not cool.

It might sound harsh, and it is (with bad punctuation usage to boot). But frankly, life is too short to live without strict black and white rules that involve sweeping over-generalizations and easy ways to get lots of people to link to your site.

Yes, there really is a point to all this.

Lots of fun today, dealing with what feels like the innards of MySQL but probably just barely scratches the epidermis. What I did learn today though was how to set MySQL system variables using a configuration file on Mac OSX. I figure that this will come up again and I’ll have to learn it all over again unless I write it down, thus, I’m writing it down while a 22 minute query runs on the other machine.

Today, I needed to increase the innodb_buffer_pool_size value from the default 8MB to something useful for my purposes like 2 GB. You can’t do that from SqlYog or from the command line as it happens to be a readonly variable. So you need to create a configuration file. That file lives in root directory at /etc/my.cnf. I first tried to create this file using vi, got it all typed up and then when I saved it, was told that wasn’t going to fly, you don’t have the requisite permissions. Stupid *nix operating system. Not that I’m complaining but after the day I’ve had, I would have liked to have just created the file.

So back to the command prompt and try sudo vi my.cnf. Lo and behold that works like a champ. The file looked like this when I was done:
[mysqld]
innodb_buffer_pool_size=2G

Saved that, restarted the MySQL server and it had updated correctly as seen using SHOW VARIABLES;

Probably all very elementary stuff but for a guy who prefers not to get his hands dirty with database stuff, good to know for the future. Also learned about profiling which you can enable in a script in SqlYog with a SET profiling=1; at the beginning of your script and a SHOW profiles; at the end.

As with most of my endeavors, I suddenly decided last week out of the blue that I needed to write a web site in Rails. I don’t really recall the thought process (which should tell you there wasn’t any) that led to this decision but given the fact that I didn’t know either Rails or Ruby past what I had learned writing WATIR scripts, I knew I needed to find a good tutorial. I reached out to my extensive (read: 82 followers) Twitter network and was pointed towards the Rails Tutorial by Karthik. Over the past 5 days, I’ve been spending a significant amount of time on learning Rails and I have to say that I have no idea why it took me this long to come around to trying it.

Quite awhile back, I dove into learning Python and the Pylons web framework and for the most part, I was happy with it. I never deployed an app using Pylons but my Python skills got to be where I could at least make things work. I enjoy the Python language but I always felt like testing was a second class citizen to a large degree. Maybe I just never read the right tutorial but tests and deployment seemed to be an afterthought. In Python’s defense, it never seemed to have the “cool kids” momentum like Ruby/Rails. And Pylons certainly has different goals than Rails given the fact that Pylons gives you some defaults but let’s you do what you want whereas Rails has the Rails Way and it’s clear you stray from it at your own risk.

But learning Pylons especially always felt like I was stabbing at zombies in the dark. I didn’t have a good mentor other than the web and while I certainly could have read code trying to figure out best practices, I’m lazy and would at least like to have some reasonable semblance of a roadmap to guide me. With Rails in general and the Rails Tutorial linked above specifically, the road map is more like a TomTom GPS system, giving you the steps you need to create and deploy applications in Rails right from the start. The emphasis on testing, source control and deployment is refreshing since I imagine that being a major headache of writing a decent sized web application in a new framework.

The use of convention over configuration, long a selling point for Rails, is wonderful. There are so many places over the last few days where I have compared my experience in the ASP.Net world to what I was learning in Rails and found the former terribly lacking. As an example, in ASP.Net MVC, named routes have to be created explicitly but in Rails, many useful named routes are created for you based on convention.

Another place Rails seems to have the advantage is database updates as you develop an application. I just went through a situation where I had to manually define how a couple of people would deal with updates to a database while working concurrently on a project. With Rails, it’s built in through the concept of migrations. So many pieces of the plumbing that you have to deal with on normal projects is just handled. I’m sure there are gotchas waiting to jump up and bite me but for someone first learning a framework and comparing it to other experiences in the past, it’s wonderfully refreshing to find so many pieces of the puzzle are taken care of for you.

On top of that, as someone who has struggled with how to coordinate programming styles and conventions on projects in the past, imagine my excitement when I learned about the Rails Way and convention over configuration. If you just do things the Rails Way, not only are certain things taken care of for you, but you can count on Rails code being comparable across projects. Imagine reading the open source code for a couple of ASP.net projects. While ASP.Net MVC has certainly moved the slider bar over towards convention, chances are the two projects would have drastically different flavors. Yet it seems to me that Rails projects would likely be reasonably similar.

Overall, I’m thrilled (obviously since I just ran through 700 words detailing my excitement) with Rails and the potential it has in my development toolbox. While it’s certainly early and I’ve no doubt been excited about things in the past, it’s encouraging to feel comfortable in a framework to some degree right from the start. Of course, major props has to go to Michael Hartl who wrote the tutorial I’m learning from. He’s done a fantastic job of laying out a good course through the material with reasons for doing things in a particular way. I intend to buy the screencasts once I work my way through the online book because I imagine they are filled with exactly the same kind of good teaching that is available in the book.

Last month at Dallas Hack Club, we did the Word Wrap Kata from Uncle Bob Martin’s “Clean Coder”. I got there a touch late and rapidly figured out that we were doing it in Ruby. Now, my Ruby skills are right up there with my Mandarin Chinese skills which is to say, I can’t even order a scotch or find the restroom so I quickly figured out that if I followed along, I’d be sitting in a puddle of pee wishing I was drunk. So I made the executive decision to ignore most everyone else and do it in parallel with the group but in Python since I can at least code like I’m drunk in Python.

My experience was surprisingly similar to the post linked above, e.g. I went down the hardest path trying to solve the wrong test. I’ve since gone back and deleted the code but the pseudocode went something like this:

Break text into a list using a list comprehension based on the length of the column
Start looking for words with spaces
Try to repiece things together based on the column length and the last space
Cry

Clearly, this wasn’t going to work. About that time, we ran out of time and went to the Flying Saucer to drink beer and pretend like we weren’t geeks. But I decided to finish the kata this week during a slow period at work. Since I didn’t have Jerry there to talk me through how dumb I was, I started reading the post up until the point where it describes writing the wrong test and spending too much time trying to solve it. So I backed up, deleted everything and started down the easier path of solving all the non-space related issues first. Once I did that, and once I figured out that recursion was going to be a big help, the project really got easier.

The final solution is below:

def wrap(text, num):

  if len(text) <= num:
    return text
  elif text[num-1] == ' ':
    return text[:num].rstrip() + '\r\n' + wrap(text[num:], num)
  elif text[:num].find(' ') > -1:
    rightSpaceIdx = text.rfind(' ')
    return text[:rightSpaceIdx] + '\r\n' + wrap(text[rightSpaceIdx:].lstrip(' '), num)
  else:
    return text[:num] + '\r\n' + wrap(text[num:].lstrip(), num)

It’s fascinating how quickly the algorithm starts to come together when you write the correct test. The problem is, I’m terrible at all this so it’s going to take some time getting enough experience to correctly select what test to write next. I was writing a ton of code trying to solve a problem that was too hard and the solution I eventually would have come up with would have been effective but brittle.

The more I do TDD/BDD, the more I realize that it is *THE* way to develop software, especially if you’re working in a dynamic language. I’m currently working through the tutorials at Ruby Tutorials and it’s great to see that TDD is a fundamental part of the process. As I learned Python and Pylons, it was up to me to figure out best practices it seemed like and that’s workable but frustrating in the long run. I’m planning on doing more katas in an effort to improve my craft.

I’ve been working on a Pylons app quite a bit lately and occasionally I run across issues that warrant documentation on the interwebs. That happened today concerning routes and how Pylons deals with them.

If you start getting an Import error that says “No module named content found”, you’ve run into it. According to the Pylons book, “Routes has a surprising legacy feature that means that if you don’t specify a controller and an action for a particular route, the implicit defaults of controller=’content’ and action=’index’ will be used for you”. This is certainly surprising to me and as always, things that are implicit tend to annoy me. Luckily, you can change this behavior. In your config/routing.py file, set map.explicit = True and then you’ll need to alter the route that is giving you problems to make the controller and action explicit. For example:

Before

def make_map(config):
    """Create, configure and return the routes Mapper"""
    map = Mapper(directory=config['pylons.paths']['controllers'],
                 always_scan=config['debug'])
    map.minimization = False
    map.explicit = False 

    # The ErrorController route (handles 404/500 error pages); it should
    # likely stay at the top, ensuring it can always be resolved
    map.connect('/error/{action}', controller='error')
    map.connect('/error/{action}/{id}', controller='error')

    # CUSTOM ROUTES HERE
    map.connect('schedule_entry', 'schedule/index/{scheduleDay}')
    map.connect('/{controller}/{action}')
    map.connect('/{controller}/{action}/{id}')

    return map

As you can see, the default behavior is for map.explicit to be set to False. The first route under CUSTOM ROUTES HERE is a named route that to my eye should match schedule up with a controller and index up with the action. Unfortunately, instead it tries to find the default implicit controller “content” which it can’t find and that throws the ImportError listed above. To fix it do this:

After

def make_map(config):
    """Create, configure and return the routes Mapper"""
    map = Mapper(directory=config['pylons.paths']['controllers'],
                 always_scan=config['debug'])
    map.minimization = False
    map.explicit = True 

    # The ErrorController route (handles 404/500 error pages); it should
    # likely stay at the top, ensuring it can always be resolved
    map.connect('/error/{action}', controller='error')
    map.connect('/error/{action}/{id}', controller='error')

    # CUSTOM ROUTES HERE
    map.connect('/schedule/index/{scheduleDay}', controller='schedule', action='index')
    map.connect('/{controller}/{action}')
    map.connect('/{controller}/{action}/{id}')

    return map

Now the map is set to explicit and the controller and action are explicitly specified which works just fine. This all may be an artifact my novice understanding of Routes but since the book documents it this way, I guess this is the way I’m going to do it.

I’ve written about test driven development and Pylons before but there have apparently been some changes to how it all works since Pylons 1.0. I didn’t run across anything in the documentation detailing the changes, specifically to the template context global and how you access it in your tests.

From the Pylons docs on testing:

Pylons will provide several additional attributes for the paste.fixture response object that let you access various objects that were created during the web request:

1. session — Session object
2. req — Request object
3. c — Object containing variables passed to templates
4. g — Globals object

To use them, merely access the attributes of the response after you’ve used a get/post command:

response = app.get('/some/url')
assert response.session['var'] == 4
assert 'REQUEST_METHOD' in response.req.environ

As it turns out, that’s sort of true. I have a test that looks like this:

from nbapowerrank.tests import *

class TestGamedetailsController(TestController):

    def test_index(self):
        response = self.app.get(url(controller='admin/gamedetails', action='index'))
        self.assertEqual(len(response.c.games) > 0, True)

and a controller that looks like this:

from nbapowerrank.lib.base import BaseController, render
from nbapowerrank.model import meta
from nbapowerrank.model.game import Game

log = logging.getLogger(__name__)

class GamedetailsController(BaseController):
  def __before__(self):
    self.game_q = meta.Session.query(Game)

  def index(self):
    yesterday = datetime.today() - timedelta(1)
    c.games = self.game_q.filter_by(gamedate=yesterday).all()

    return render('/gamedetails.mako')

Unfortunately, contra the documentation that says the “c” alias will be available on the test response object, that test always fails with an AttributeError stating that in fact, the c attribute does not exist. It frustrated me even more because all the other attributes that are supposed to be on the response like session and g were in fact there. After doing some random digging, I came across this in the Pylons 1.0 roadmap: “Deprecate pylons.c, pylons.g, and pylons.buffet. These have been disrecommended since 0.9.7.”

Apparently, they are/have deprecated using the c alias for tmpl_context even though when you create a controller under Pylons 1.0, it still aliases the context as c. Sigh. So in order to test data that you have added to your context for templating, your test should use the explicit tmpl_context instead of the c like this:

from nbapowerrank.tests import *

class TestGamedetailsController(TestController):

    def test_index(self):
        response = self.app.get(url(controller='admin/gamedetails', action='index'))
        self.assertEqual(len(response.tmpl_context.games) > 0, True)

And then all will be well with your world. I do have to say that compared to both ASP.Net MVC and Rails, the testing support in Python/pylons seems to be a second class citizen. I’m not sure why that is because as a dynamic language, it seems to benefit greatly from a TDD approach. Maybe it’s just me getting back into a framework after a year’s worth of changes.

Anyway, that may help some people out there trying to search in the googleplex about TDD and Pylons.

With my deepest and sincerest apologies to Elizabeth Barrett Browning, God rest her soul.

Original poem obviously by Elizabeth Barrett Browning

Global Assembly Cache, How do I hate thee? Let me count the ways

How do I hate thee? Let me count the ways.
I hate thee to the depth and breadth and height
My soul can reach, when feeling out of sight
For the ends of Being and ideal Grace.
I hate thee to the level of every day’s
Most quiet need, by sun and candle-light.
I hate thee freely, as men strive for right;
I hate thee purely, as they turn from praise,
I hate thee with the passion put to use
In my old griefs, and with my childhood’s faith.
I hate thee with a hate I seemed to lose
With my lost saints -I hate thee with the breath,
Smiles, tears, of all my life! -and, if God choose,
I shall but hate thee better after death.

I’m currently reading ASP.Net MVC 2 in Action and overall, the books seems solid. I can say this because I’ve read about 20 pages and agree with most of it. However, the 20 pages I’ve read do contain some advice that seems overdone at best and downright confusing to future developers at worst. The chapter I’m reading is Data Access with NHibernate. I’m working on an application that contains an ASP.Net web site backed by a PostgreSQL database. Previously, all my applications used MSSQL and therefore were set up using Linq-to-SQL as a poor man’s ORM. With PostgreSQL, that’s no longer an option so I’m in the process of learning NHibernate and Fluent NHibernate, a task that’s long overdue.

I hate learning a new technology by doing everything wrong the first time so I went looking for some best practices or architecture suggestions for setting up NHibernate. This book had an entire chapter on that and so I dove right in. Overall, it’s been very useful. Heavy use of Dependency Injection and Inversion of Control nicely decouple the pieces of the app from each other. However, the authors recommend something that seems a little extreme to me.

The example solution has a UI project which is the ASP.Net site, a Core project containing domain models and code, an Infrastructure project for things like data access and assorted test projects including an Integration Test project. The authors point out that the only project that references the Infrastructure project is the Integration Test project. Their rationale for this is that Infrastructure is necessarily fluid. Because of this, you don’t want to couple the core or UI to it. They set this up by using runtime DI to inject dependencies from the Infrastructure project into UI components. Specifically, the data access repositories that certain controllers need are discovered at runtime using settings in the web.config. They claim that this results in a completely decoupled application.

However, in order for this to work, the UI project needs to have access to the Infrastructure assemblies and config files. Normally, this would happen via an explicit reference in Visual Studio which would result in the necessary files being copied into the UI project at compile time. Because the UI project doesn’t have that reference, the authors have to get the files there another way. Their solution is to use a post build step in the Infrastructure project to copy the necessary files. To me, this only serves to make the reference implicit, something is likely to cause issues down the road.

The UI definitely has a dependency on the Infrastructure project. It seems extreme to hide that dependency in a post build step instead of showing it explicitly in the project references of the website. It’s one thing to write decoupled code that is easy to test and change. It’s entirely another to force developers to jump through hoops and keep track of idiosyncrasies like implicit project references. However, being a complete newbie to this form of architecture, maybe I’m missing something. Is there a true reason for managing dependencies between projects in this manner? If so, why not manage all of them in the same manner, do everything at runtime and copy all files via post build steps? I think the answer is that there isn’t a real reason for this except for the purity of the architecture, something that should immediately be questioned for intrinsic use. I’d love to hear any opinions from the experts out there.