Biomime: Coding and Biology

Questions for an Interview

2009-10-19T18:37:00.000-07:00

QUESTION: Based on your resume, you have [seen lots of different management styles OR had extended experience with a single employer]. With that experience, I'm sure you've seen your share of good and bad management practices. What are some of the worst things you have seen management do? What style of management works best for you?

Of course, this is a trap! This is a good indication of whether or not the person will fit into the management style you use. Typically, the applicant will answer "micro-management" (as the worst style), and that they prefer someone a little more hands-off. ...I find this to be a cop-out. A good answer here would use a specific example of something that management did poorly and how the applicant attempted to resolve it. The best answers to "best for you" would involve managers that give the applicant the tools they need to succeed. Such answers demonstrate that the person knows how a team actually works, that they are driven, and that they can overcome obstacles.

QUESTION: What do you think are the most important qualities of [good code]?

...Note that this could be applied to pretty much any trade, just be sure you're asking about the resulting product, not the position--that's a different question.

Any answers that demonstrate they have thought about this (hopefully, a LOT), and they've formulated their own opinions about what makes the product good. of course, you can choose to add your own interpretation to their answers to see if they are a good fit with your company, too: do they share values with the other workers?

QUESTION: What do you think are the most important traits for a [developer]?

...Note that, of course, you can change "developer" for anything you want.

You want them--WITHOUT PROMPTING--to list off their own best traits, and to explain how they posses them. If you have to prompt them with a question like "...and how do you think you measure up to these?", then you've found an applicant that isn't very applied and is probably low-confidence. It comes out naturally with good applicants. And, again, you want to compare their answer with the values of your current employes to see if it's a good match.

QUESTION: What makes you a remarkable candidate? What do you think makes you stand out from the other applicants?

Well, you may or may not want to see some humility here ("I don't know if this makes me *remarkable*, but..."); that's up to you. But this answer yields a good idea about their level of confidence and what they like to do most. ...Not necessarily what they're best at, since their answer is so biased. : )

QUESTION: Out of all the people you've worked with, who would you say you respected most, and why?

[after they answer]

...If I were to ask that person to describe you, what would they say?

Well, I'm not sure, but the point of this question is to get the applicant to think about two things and how they relate: what they like most in a co-worker, and what co-workers probably think of them.

QUESTION: Assuming you get this job, what would be your biggest fear? What do you feel could go horribly wrong?

If nothing else, I think this question gives you a chance to see how the applicant handles a stressful, difficult question. Ideally, they would answer honestly, and tell you what the biggest potential incompatibility would be. However, I find that most applicants shrug this question off and use it to make a joke. I've also seen applicants really choke on it, which (to me) was a good indication that they weren't likely to work well under stress, and that they aren't comfortable being honest with authority figures.

Kids' contest: art and nature

2009-10-15T06:01:00.001-07:00

http://www.gettoknow.ca/us/contest/

The contest, recently launch in California for the first time and currently accepting entries, is an invitation for youth (age 18 and younger) to go outside (or to a zoo, aquarium, or museum), to "get to know" their wild neighbors, and then to share their experience with the world by creating an original work of art, photography, or writing and then entering their creation in the contest.

...Perhaps something EOL could get involved with? Sounds like it's CA-only at the moment, but I could be mistaken.

Legacy Doesn't Always Mean "Old"

2009-07-16T14:51:00.000-07:00

I've decided the EOL website development project is a "Legacy Code" project.

That may seem a bit naive, since it's really a start-up project that's two years old, give or take. However, there are three reasons for my decision.

Because it was a rapid start-up project that was done without "real" Agile development. It accrued a surprising amount of code debt in its early days.
Because the database is non-Rails in some serious ways. Particularly in its use of compound keys and the fact that it's spread over three databases instead of one. There are good business reasons for these decisions, but they affect our use of Rails.
Because of the lack of robust testing.

It's the latter case that I am most interested in addressing first and with highest priority. Our team agrees, but also points out that there is a lot of work needed in the area. We have lots of tests! But they are difficult to use, fragile in some cases, slow, and so on. Given these problems, we had a 90 minute meeting today, where we asked four questions:

What are our overall testing goals, really?
How well are we achieving them, currently?
What are the most pressing needs?
How can we best apply our time to improve tests?

I found these to be very effective! To answer the first one, we developed the following three answers:

Facilitate careful development (TDD).
Prevent regressions from being deployed.
Create enough trust to allow refactoring.

To the next three questions, we had a lively and effective discussion, with lots of excellent ideas and approaches discussed. The end result was a list of concerns, another list of "action items" to address the most important among them, and, interestingly, a list of "philosophies" about how to approach tests. They are somewhat specific to EOL, but they were cool enough that I feel compelled to share them:

View and controller tests are really much like "unit" tests, since they are testing a method on a class... it's just a different "kind" of class.
It's okay to throw away unit tests (including controller/view), as long as the features they are supporting are all tested with higher-level tests, and those tests are passing. Implementation methods change, some of the requirements of the past are likely to change, too.
Try to write tests that are not fragile (meaning: refactoring doesn't fail tests). The underlying idea is that when a feature is complete, all tests should pass and when the feature is incomplete, some tests should fail. Period. The interals are less important, though someone may have a need to test them (at the Unit level) to help with developement.
The blackbox, factory approach is more appropriate than mocks and specs in EOL's environment. But we need to make the domain logic approachable to developers (new ones in particular). Copy/pasting solutions from other tests /works/, but is not ideal. We need to make this easier to use, more convenient. Developers should /want/ to write tests, becaise it's easy to do. Newcomers have varying levels of domain knowledge.

On Creating Effective Web Interfaces

2009-07-10T17:03:00.000-07:00

At EOL, we've been asked to implement some new features that we (the developers) didn't like. They didn't seem to adhere to web-standards, didn't look particularly good, and didn't really seem to capture the problem (caveat: as we see it) elegantly.

This sparked a discussion (healthy, not heated) about how we could best approach design and web usability as we add features. Another developer wrote a blog-post about it. (...internal-only; sorry, no link for you!) I felt compelled to talk about the same subject myself and field some ideas. His post was all well-formatted, edited, and technical. Mine is casual and sloppy. So it goes. ; )

So what's the problem?

We're using Agile methods, so the proximal problem--which we are already trying to address--is that the temptation is strong to just "get something out there". Users can respond to it, and we can quickly adjust our approach to build something they will like better.

This has a few negative consequences:

The code gets ugly and risks leaving remnants of old styles when it changes. Have a look at our CSS, for example. :(
I think users can get scared away. At least for me, I know I've tried to use some feature of, say, Yahoo! Mail, hated it, and never looked at it again. ...They may very well have cleaned things up later, but I don't care to look.
User feedback tends to... well... suck. In my experience, they'll ask for things that actually result in clutter. Also, it's only the squeaky wheels that get heard, and their needs may be vastly different from the majority.
There is a big but poorly-understood difference between a feature that's easy to use and one that is "intuitive". The latter implies that a newcomer can step in and use the tool... well... intuitively. This is great! But it often creates an interface that wears on your quickly, or ends up something you hate over time. Photoshop's clone tool is a good example of a tool that's easy to use, but not at all intuitive. It's important to know which is better for your problem. I think that user-generated feedback generally comes from users who have tinkered with an interface for something on the order of minutes. They're talking about intuitive use, not easy use.

...There are probably others, but I'm getting bored. ; )

The core problem, I think, is that we really need someone skilled at web interfaces. But we don't have the money to add someone to the team to do this, and everyone currently on the team is too busy to pick up the skill and/or apply it at the expense of their current duties.

In short, we are left with only one option: establish some guidelines by which we can do the best we can, on our own.

To that end, I like the idea that my associate mentioned (interestingly enough, by quoting yet another associate of mine): a list of usability questions that can be asked to gauge whether an interface works or not.

...I think this is a reasonable start. That said:

I don't know who we should ask these questions.
I worry that we'll end up with design-by-committee interfaces, which usually suck. Ideally, I hope that design can be limited to two people on any given feature. ...Ideally-ideally, the same two people on every feature, to maintain consistency.
Nothing in the questions mentions web standards/best-practices.
I worry that this is going to take up more time than we have, and breed frustration at the slow implementation of new features.
I think the questions are good for measuring usability, but there are other concerns, such as intuitive-vs-easy, elegance, and the "message" conveyed by each feature. I hope we can add questions into the mix so that these things are considered.
I think it's worth investing two or three people's time over a day or two (each) to research usability, and contribute to the "checklist" that we're building. This could include best-practices, design methodology, questions to ask, and the like.

I could go on. This is a fascinating subject with many different possible solutions. ...Sadly, I don't think any of them are even remotely as good as "hire an expert". : (

"rake stats" should count all my stuff!

2009-06-02T19:01:00.000-07:00

We have some custom directories for Specs, which aren't counted when you run "rake stats". I thought we deserved more credit! : )

I put at the bottom of our Rakefile. It's a little... brute-force, as it trumps RSpec's own task of the same name, but it does the trick:

# We have some pretty customized stat directories, so:
require 'spec/rake/spectask'
namespace :spec do
  # Setup specs for stats
  task :statsetup do
    require 'code_statistics'
    ::STATS_DIRECTORIES << %w(Blackbox\ specs spec/blackbox) if File.exist?('spec/blackbox')
    ::STATS_DIRECTORIES << %w(Model\ specs spec/models) if File.exist?('spec/models')
    ::STATS_DIRECTORIES << %w(View\ specs spec/views) if File.exist?('spec/views')
    ::STATS_DIRECTORIES << %w(Controller\ specs spec/controllers) if File.exist?('spec/controllers')
    ::STATS_DIRECTORIES << %w(Selenium\ specs spec/selenium) if File.exist?('spec/selenium')
    ::STATS_DIRECTORIES << %w(Helper\ specs spec/helpers) if File.exist?('spec/helpers')
    ::STATS_DIRECTORIES << %w(Library\ specs spec/lib) if File.exist?('spec/lib')
    ::STATS_DIRECTORIES << %w(Cucumber\ features features) if File.exist?('features')
    ::CodeStatistics::TEST_TYPES << "Model specs" if File.exist?('spec/models')
    ::CodeStatistics::TEST_TYPES << "View specs" if File.exist?('spec/views')
    ::CodeStatistics::TEST_TYPES << "Blackbox specs" if File.exist?('spec/blackbox')
    ::CodeStatistics::TEST_TYPES << "Selenium specs" if File.exist?('spec/selenium')
    ::CodeStatistics::TEST_TYPES << "Controller specs" if File.exist?('spec/controllers')
    ::CodeStatistics::TEST_TYPES << "Helper specs" if File.exist?('spec/helpers')
    ::CodeStatistics::TEST_TYPES << "Library specs" if File.exist?('spec/lib')
    ::CodeStatistics::TEST_TYPES << "Cucumber features" if File.exist?('features')
    ::STATS_DIRECTORIES.delete_if {|a| a[0] =~ /test/}
  end
end

RCov Gem Installation problem

2009-05-15T07:44:00.000-07:00

I couldn't get RCov installed this week:

$ sudo gem install rcov --trace
ERROR: While executing gem ... (OptionParser::InvalidOption)
invalid option: --trace

Poking around a wee bit, I discovered that this was just rubyforge being obnoxious. Go download the tarball form of the file and then just "sudo gem install [name of the tarball]". All good.

Testing Your WebApp

2009-03-20T16:45:00.001-07:00

I've been looking into the various ways to test web applications, including performance, acceptance tests, and cross-browser compatibility. In my travels over the past week-and-a-half, I have decided there are three levels to this:

Testing the design of your site. Does it look similar enough in all supported browsers?
Testing the function of your site. Does all the Javascript work? Is everything that needs to render rendering?
Testing the reliability of your site. How fast is it? How many users can it handle? For how long?

Design Testing

A good first step is to validate with the W3C tools and try to stay 100% standards compliant. But you're doing that already, I'm sure. : )

Adobe has a neat "Meer Meer" application that's coming... sometime. There is also SuperPreview. Both of these allow cross-browser design using nothing more than whatever web-browser you have installed: they call the real browsers on the server-side, take snapshots, and serve you the images. The coolest feature here is that you can overlay two browser's snapshots and have the differences jump out at you. ...Both of these are unreleased, but just around the corner. Very cool.

If snapshots are enough, you can get these cheap: $50 a month for Litmus. You can hook these tests into your testing suite. CrossBrowserTesting has a demo video here. Nothing earth-shattering: just note that you have to do these "tests" by hand.

IBM says: "For instance, for a monthly fee, Browsercam.com gives you access to a broader range of platforms and browsers—including some mobile platforms—and these pages are dynamic, allowing you to test Ajax apps and other JavaScript-based pages. Some services, like Litmusapp, can integrate with your testing and development workflows, offering versioning support and ways to publish test results to external clients."

Interestingly, if you want to do these things by hand, there are free Microsoft images of common Windows versions/Browser combos for use by developers. The bummer is that you have to renew them every so often (and only supported OS/browsers are represented), but it's a great way to test your app without spending too much money on setup.

A decent rundown of the "images of site" tools is here; (there were other overviews, not as good) we should see what the JavaScript-capable options are really capable of. If you need older versions, you'll have to install those yourself, but old IE versions are still available on the web (not from Microsoft, however).

Functional (Acceptance) Testing

For acceptance testing your Javascript, there are a few choices in the world of headless (ie: much faster) browsers. HtmlUnit seems a usable Java "mock" browser. There's a JRuby wrapper for it called Celerity, which could get me started faster. Similar is jsTest, also in Java. Jiffy struck me as overly complex, and hard-wired to Oracle for measures. Joe Dog Siege looks like a usable, mostly command-line driven load tester.

An interesting but probably less-useful alternative is to just check the JS for problems using crosscheck, but as of now it doesn't seem to have usable documentation, so...

If you don't mind (or have a need to be) using real browsers, Selenium is the standard right now for writing acceptance tests, and it's really easy to set up. Whether it adds overhead is an interesting question; I would assume yes based on some comments from other tool's sites. That said, A) You don't want to mix your acceptance tests with your performance tests, and B) it seems to be a standard right now, and I do recommend using it because of that. There are some alternatives, though. For example, JsUnit is a way to do the tests in native JavaScript, which is quite possibly useful.

Performance Testing

"Performance testing" (actually lots of sub-categories; read on) is a rather large arena, and I found this stuff quite interesting. ; ) I found the best introduction to the topic to be this neat video from Google, which gives some definitions:

performance testing - how fast will it respond under load X? You need to test things at least long enough to reach a steady state. Start up slowly: don't just throw 100 users at it, build it up over a warm-up period. The machines under load periodically push performance dat (see blow) to monitoring servers, which analyze and store the data. These monitoring servers are also monitoring themselves (so you know what the effect of storing the data is). To the extent possible, remove the network: run everything in its own little environment.
stress testing - at what point does the system break under load (and does it do so gracefully)?
load (or liability) testing - given a high load (80% of max) for an extended period of time (Google does 1 month, 72 hours is a minimum recommendation), how does it perform? Often there are cron-related or system-level tasks (ex: log rotation) that affect performance only at certain times, so this needs to be examined.
benchmark - a simplified measurement that is repeatable, representing a typical customer's activity on the system.
profiling - When you change some aspect of the system (memory, load balancer, etc) and measure the differences in benchmark performance.
availability testing - when a system fails, how long does it take to come back up? Google operates under the assumption that their equipment sucks, so they will solve that problem in software.

Her reasons for testing are 1) to make honest claims to the client backed by data, 2) test hardware changes, and 3) developers can see the effect of their code over time. NOT for functional testing.

Some other tips included: run every test at least 5 times before you measure it: inconsistency must be accounted for. She recommends a single five-minute test run "on demand", when a developer requests it, because you get *some* information from it, but nightlies are still required.

She recommends JMeter to measure loads (but it doesn't handle JavaScript) (There is a demo of how to get started, too), or Grinder (for "APIs").

Note that you should track not only the QPS, response times, etc, but also the CPU load, memory, versions of every piece of software on the machine, and disk i/o.

Interestingly, she says that Linux is typically performant as long as you don't exceed any single resource's 80% mark.

You can buy load testing. Prices on LoadStorm, for example, range from $30 to $200 a month. LoadRunner is another alternative (software that you use), by HP, which appears to be rather expensive... (Well, when they don't list a price, one must assume...)

BrowserStorm looks reasonable for a paid service, and has an explanation of its features in a blatant plug on Ajaxian.

Hope this helps.

AIM Chatroom Death

2009-03-02T07:53:00.001-08:00

...So we EOL developers hang out in an AOL chatroom. Last week, I got logged out of it, and couldn't get back in. ...Not sure why. I hoped the problem would just go away, but it never resolved itself. The error I was getting was:

Couldn't join chatroom "".

An AIM service error occurred.

Error: Serv:RequestTimeout

ChatError:NoError

Useless message! But eventually I found a page describing the problem, with the suggestion to change the AIM server port from 5190 to 443.

That worked.

Fixing "svn: Checksum mismatch"

2009-02-27T07:01:00.001-08:00

While trying to check in some changes I made, I ran into a problem that gave me the following error:

Transmitting file data .svn: Commit failed (details follow):
svn: Checksum mismatch for '/Users/jrice/workspace/eol-dev/spec/models/.svn/text-base/taxon_concept_spec.rb.svn-base'; expected: 'e4bd7ddf1f3c78447253dbdcd4ab96f6', actual: 'edbe4f2e08778a7d4f33212a812450ec'

...At first, I really freaked out, wondering if I had some kind of virus tacking on additional information to my files or something like that. But, of course, that was just panic. In truth, this is just a bad MD5 checksum in the svn database, and only for one file. It's kind of a pain to fix, but it's simple enough.

I found a very nice walkthrough to fix it on Chris's Glob blog. For posterity, I will re-cap it here. Start by committing everything you can (basically, everything but the one file that is having trouble). Here's the rest of what I did, based on Chris's walkthrough:

[models]
$ cd /tmp
[tmp]
$ svn co https://repository.eol.org/eol/trunk/spec/models
A    models/toc_item_spec.rb
A    models/taxon_concept_spec.rb
A    models/transactions_spec.rb
A    models/normalized_names_and_links_spec.rb
A    models/data_object_tag_spec.rb
A    models/hierarchy_spec.rb
A    models/user_spec.rb
A    models/factories_spec.rb
A    models/hierarchy_entry_spec.rb
A    models/name_spec.rb
Checked out revision 295.
[tmp]
$ cd -
/Users/jrice/workspace/eol-dev/spec/models
[models]
$ cd ..
[spec]
$ mv models /tmp/models.orig
[spec]
$ mv /tmp/models .
[spec]
$ svn up
At revision 295.
[spec]
$ cp /tmp/models.orig/taxon_concept_spec.rb models
[spec]
$ svn st
M      models/taxon_concept_spec.rb
[spec]
$ svn commit -m "Adding specs to taxon_concept in prep. for xml API"
Sending        spec/models/taxon_concept_spec.rb
Transmitting file data .
Committed revision 296.

That's it: as you can see, the file went up just fine after that.

Testing [warning: LONG post]

2009-01-23T13:54:00.001-08:00

So, today, Remi (of OpenRain) and I made a presentation to the other developers about what we thought would be the best M.O. would could pick for testing our application. Without going into the "whys" behind this testing sea-change, I thought I would recap what I've learned, with Remi's help.

First, we've re-defined what to test. When our testing suite was at its low point, the answer to that question was "everything". ...And that's what made the tests so brittle: if anyone made any change at all, the tests let them know about it. So, coming up with a clear definition of what gets tested was important to me. The answer I chose was twofold:

Those parts of the code (namely, APIs) that really should never change, and
The functionality that we really need to prove works.

The latter point is made particularly clear by the argument for black-box testing. The most poignant case he makes in that video is to make the assertion that "this really doesn't prove anything", and that was very important to me.

So, with those two core definitions, we decided to take two core approaches:

Unit testing
Full-stack (or nearly-so) functional testing

...That is to say, no view tests: they depend too much on everything else to render properly and are too difficult to test in ways that aren't going to break when you decide to, say, rename a div. Secondly: no controller tests. As Remi put it, controller tests are huge tests, and they prove almost nothing. They are not worth their weight.

I looked into a myriad of tools for writing tests, including shoulda, context, matchy, zebra, and a review of RSpec. In a nutshell:

Shoulda is really cool. The assertion that one line of Rails code should require one line of testing code is... superb. I'd like to work under that assumption, and write our own custom assertions to facilitate this. But shoulda also relies heavily on an application being very much in line with the Rails Way, and our app, because of its highly complex, stand-alone database is not particularly so. I feel much of the power of shoulda would be lost on us. So, rather than learn this system, we should just pass it by. Fortunately, RSpec has made some changes to their system that make shoulda-macro-like custom assertions and "one-liners" much easier.

Context and matchy are also neat, in that they give you much of what RSpec and shoulda give you, with much less baggage. I like that concept! But when I actually installed the code and tried it out, a lot of the things that "just work" in RSpec just... didn't. Rather than wrestle with it for more than a day, I decided these tools are... well... too light for our purposes.

Zebra impressed me the most. I think we'll all be writing tests in this style (example: expect { @my_model.to be_invalid() }) in the near future. ...But at the moment, one needs to define too many of those assertions one's self, and thus I don't think is is really mature enough for us to start using without serious investment in time. ...which we just don't have.

In the end, I decided it was wisest to stick with Rpec. It's got everything one needs to write excellent unit tests... and then some. So, yes, it comes with some overhead and baggage, but on the other hand: the things you need to "just work"... do.

That said, the style of testing we'll be doing needs to change. As I said earlier, we should focus on testing only those parts of the code that really shouldn't change. For example, we expect a taxon_concept to have a common name, which comes from a particular part of the database and defaults to the scientific name when that entry isn't found. These are things we can test, and we can do it without stubbing the tar out of every method that ever gets called to create the end result.

Consequently, we will be hitting the database to run tests. And this makes me a little sad, because I know how slow testing can be when it's dependant on the DB.

To alleviate some of this, we are going to try and stop using fixtures in favor of factory_girl. This has several advantages:

Fewer models than with fixtures. If you need to test a "special" model, you instantiate one with the special feature. It's not there in every other test that doesn't need it. Hopefully, this will cut down on the time it takes to prepare for any given test.
Easier to define (DRYer) than fixtures. There's (nominally) one factories.rb file sitting somewhere, with all of your models defined in a rather succinct syntax. Compare this to the 50+ YML files sitting in a directory. If you tweak the relationship between two models, you're not doing a search-and-replace on several YML files, you're changing one definition in one place.
More robust than mocks. The problem with mocks is that you need to stub each function that gets called on them, and this can be quite expensive (in developer's time) and non-dry, if not done carefully.
More coverage of class behaviours. So, when you call that name() method which bounces all over creation to find your common name, you're flexing the muscles of all the pieces involved to make sure they work. Of course, this "feature" comes at the price of less isolation of code. ...and isolated tests is one of the hallmarks of RSpec. But I think it suits our project better to take the coverage. Plus, our project makes heavy use of find_by_sql, which may otherwise go untested.
Easier to instantiate than mocks. Our "top-level" model, a taxon_concept, relies on around 20 other models to actually work. With RSpec mocking, I had to create each of those mocks and tie them together. The resulting code was very, very ugly. Yes, I probably could have cleaned it up, but I don't think I could have gotten it nearly as succinct as factory_girl's syntax.

Factory_girl is a rockin' module. I think everyone should be familiar with it. (And, yes, I am aware that there are a number of viable alternatives with the same underlying behaviour. But f_g seems most popular and least cluttered.)

...So we still have the problem of proving that the website works. This was a problem with RSpec. Because of its fantastic isolation of testing, one was never really sure if the whole stack was going to behave properly. I spent the vast majority of my time in the past 1 1/2 weeks trying to solve this problem of proof that the damn site actually does what you want. There are plenty of solutions out there, but personally, I found most of them clunky. ...at best.

Enter webrat. This is a package that makes visiting a site as simple as... well... visit(url). And you can click around, fill in forms, and all of those similar things with other, very simple syntax. Example from the current homepage:

  class SignupTest < ActionController::IntegrationTest

    def test_trial_account_sign_up
      visit home_path
      click_link "Sign up"
      fill_in "Email", :with => "good@example.com"
       select "Free account"
      click_button "Register"
    end

  end

...Isn't that slick? ...This makes writing good functional tests a piece of cake. And, while we could easily run these kinds of tests with RSpec, I decided that, because we operate on a user-story-centered style of implementing features, I thought we could also adopt cucumber, which is another really slick wrapper around user stories. Basically, you write tests in plain english, using Given / When / Then blocks, and write some ruby code to match your plain-english assertions and turn them into webrat (or some other) full-stack tests.

I expect some resistance to cucumber. It feels a little... hokey to write tests in plain english, then parse them in ruby... but in practice, I have found the technique to be very readable, very usable, and surprisingly minimal (in terms of the amount of code). Assuming there is sufficient buy-in for this, I actually believe it will turn out to be a really cool, really reliable way to, as I said, prove that the site works.

Of course, all of this is academic as of this afternoon. We'll see how things pan out in the next week or two. Some of these ideas may not be well-conceived, or may turn out to be ill-applied to our particular codebase. I'll keep open-minded about it. But I'm also really excited to at least try and get all of this to fit together nicely.

I actually rather enjoy writing tests, and I think these changes will make tests more fun, more useful, and more productive. We'll see.

"Errors running test:units!"

2009-01-23T07:36:00.001-08:00

Sounds nasty, doesn't it?

Worse, there's little clue, even with --trace on, as to why it failed.

In my case, however, I poked around a bit and noticed that my "test/test_helper.rb" file was missing. (Long story as to why.) Restoring it fixed the "Errors running test:units!" problem.

Hope that helps someone else out there ('cause I couldn't find a good explanation online)...

config.gem 'rspec'

2009-01-22T10:43:00.001-08:00

I finally got rspec to install using config.gem (in Rails) with these lines:

config.gem 'rspec', :lib => 'spec'
config.gem 'rspec-rails', :lib => false # Note this hack keeps it from failing on this gem.

...That allowed me to install/unpack them. Which is all well and good. But when I ran my specs, it failed, complaining about an "undefined method `evaluate_value_proc' for class `Spec::Matchers::Change'". Looking around, the claim was that this error was caused by "incompatible versions of rspec and rspec-rails". But that didn't jive with the versions I had installed.

So I commented out my two lines, above. ...At least I had them in vendor/gems, (and my "script/generate rspec" bootstrap command ran fine, earlier).

With the gems commented out, everything worked.

[shrug] Go figure.

Updating to RubyGems 1.3.1 on OS X

2008-12-16T08:11:00.001-08:00

As much as I enjoy ruby and rails, I don't keep up with the community much.

This has its price... sometimes I'm blindsided by things that I imagine are "common knowledge" on the mailing-lists, but aren't documented anywhere that we lesser mortals actually notice.

The problem I had this morning was mentioned, in obfuscated form, on one blog, at least. I thought I would state it more explicitly:

I updated to Rails 2.2.2, but it complained that I needed to update to RubyGems 1.3.1. When I tried (sudo gem update --system), I got this message:

Updating RubyGems
Nothing to update

...It turns out you need to use the "alternate install" for RubyGems. ...despite the fact that the documentation says explicitly that you won't need to if you have a more recent version. Do this:

> sudo gem install rubygems-update
> sudo update_rubygems

...Now you're running 1.3.1, and Rails will be supplicated.

Bugs Me:

2008-10-22T09:26:00.001-07:00

script/console test
mongrel -e test
 rake db:migrate RAILS_ENV=test

Why?

Consistency is a good thing.

IE6

2008-10-14T07:24:00.001-07:00

I generally don't pay attention to server stats, but recently I was getitng frustrated with IE6, so I asked those in the know how prevalent IE6 was. The reply:

"...as a total, IE 6 is about 22% of all users"

My response: holy shit. That's obscene! IE6 is broken, and to think that over 1 in 5 people are still using that crap-tastic browser is an offense to the internet. Sure, sure, I can understand needing to keep IE6 on some fragile server somewhere that, if you update it, your entire site breaks. Fine. But stop browsing with the damn machine! There's no good reason to keep using IE6.

One in five!!!

People: switch!

Composite Primary Keys (everyone's got an opinion)

2008-09-17T20:29:00.001-07:00

Tonight I was playing around with some of the "cooler" things that Rails can do, and I thought I would try them out on our code.

Namely, there are two features that I'd like to use: nested_has_many_through, and named_scope.

This would allow us to write code like "taxon_concept.data_objects.images.visible" without having to go through a hairy find_by_sql, and it would allow us to more easily make changes to the DB (like, say, how we define "visible" for data_objects). It would have made one of today's changes painless, too (long story).

It would also reduce the amount of code we have by a significant amount. ...I like less code.

...The problem, though, is that DHH doesn't understand composite primary keys (CPKs) *at all*, and his framework reflects that. Even with the CPK plugin, things don't work right. ...That is, when you declare a CPK on a model, it seems that it expects every reference to it to contain all of the PKs. I tried doing something like (pseudo-code): "DataObject.has_many :data_objects_taxa, :foreign_key => :data_object_id ; DataObjectsTaxon.has_many :data_objects, :foreign_key => :id" (DataObjectsTaxon has a CPK)... but this doesn't work: it STILL tries to pass in two key values. The resulting queries are actually rather asinine (notice the 1,4 at the end), and fails miserably:

    SELECT * FROM `data_objects` WHERE (data_objects.id = 1,4)

...I could delve into the CPK plugin code and try to resolve this, but I might be gone for days. ; )

You can kind of get around this by using a habtm relationship. ...But that only works for that one hop. So I can get data_object.taxa, but I can't get data_object.resources, using this code:

    has_and_belongs_to_many :taxa
    has_many :resources, :through => :taxa

This throws "Unknown column 'taxa.data_object_id'"... and the docs even warn about this ("You can only use a :through query through a belongs_to or has_many association on the join model").

So.

What I would like to do--knowing full well that this is the wrong thing to do, but bending to the will of Rails--is change our CPKs to Unique Indexes, and add autoincrements to those tables. I could write the migrations to do such things without too much trouble, but I worry about the affect it would have on the related PHP code (I hope none, since the autoinc should just merrily do its thang).

Again: from a database level of understanding, this irks me. Where we use CPKs is where they are uniquely identifying entries, and that's how its sputta be. ...but Rails thinks otherwise.

...Unless someone knows a way to keep things as-is and get the has_many relationships working in both directions. There my be some trick to it that I'm unaware of. That would be cool, too.

If we could get this to work, I think our code would be much, *much* more elegant (and thus, maintainable/extensible). I'd be happy, assuming there isn't a noticeable performance hit (but from what people are saying, it's actually quite efficient).

Thoughts?

And: ...apologies to California Pizza Kitchen (CPK).

Murphy's Law of Variable Names

2008-09-17T13:43:00.001-07:00

There are at least six appropriate names for any given variable, method, column, or table.

You will not choose one of them.

You might be a geek if

2008-09-10T12:23:00.001-07:00

..."too much rails" sounds grammatical to you.

Composite Primary Keys (CPK) and RSpec Fixtures

2008-08-01T10:17:00.001-07:00

I was banging my head against the proverbial wall all week. This morning's travail was an error like this in my specs:

You have a nil object when you didn't expect it!
You might have expected an instance of Array.
The error occurred while evaluating nil.first
/path/to/rails/app/vendor/gems/composite_primary_keys-1.0.5/lib/composite_primary_keys/base.rb:239:in `find_from_ids'

...After some digging, I found a page that explained the problem, and revealed that you need this in your spec/spec_helper.rb file:

load 'composite_primary_keys/fixtures.rb'

I feel much better now.

Repetitive Stress

2008-07-29T07:25:00.001-07:00

So, I've been programming pretty much non-stop for the past few months. My wrists ad neck were suffering.

Rather than start wearing braces again, I opted to try this program (Mac-only) that interrupts you every so often. ...I have it set for 12 seconds every few minutes, and 10 minutes every hour. I started using it about a month aogo

It seems to have done wonders for my hands and wrists, so I highly recommend it.

Ain't done squat for my neck, though. ...And some days, it gets really bad. : \

Open Source, Sickles, and Hammers

2008-07-23T13:24:00.000-07:00

(Image cc-by jagelado)

Using Vim and MySQL to Create Fixtures (for Rails)

2008-07-02T14:57:00.001-07:00

...I've had to build a lot of fixtures using extant data from a MySQL database. I don't want the whole database in my fixtures... just a subset, based on some examples I use in my specs.

So I run a query to get the subset I want, being sure to grab all the contents from a given table (and ONLY one table). Something like this:

SELECT st.*
FROM a_fancy_table ft
JOIN some_table st ON (ft.id = st.a_fancy_table_id)
WHERE ft.id IN (1, 2, 3)\G

The \G is essential, here. I then copy the output of that to the some_table.yml file, created with vim.

Then:

<ap multiple times, until everything is left-justified, then
>ap once, so that it's all indented 1 level, then
:%s/ \*\+ $\d\+$. row \*\+/^Msome_prefix_\1:
finally, rename any of those that are really significant.

...Of course, the ^M is really ^v[ENTER], and the some_prefix is whatever you want to name your objects.

...If that makes sense, you'll find it's a huge time-saver. Unless you're copying entire tables, in which case you can use Dmitry's script.

ZenTest and autotest in Windows with RSpec and Rails

2008-06-12T09:05:00.001-07:00

I've been trying to monkey-patch ZenTest's autotest for over a month now. Originally, I was trying to get it to work (well) with snarl, but that just seems frivolous at this point. My focus now is just to get the damn tests working as they should.

First of all, Windows doesn't let you break out of the loop. ...At least, not that I've ever seen. I haven't heard a SINGLE person on the web talk about this, which is frustrating. Maybe I'm missing something stupid, but I know other people that this is happening to.

Anyway, the fix for that is to change the "Kernel.sleep 1.5" on line 291 of autotest.rb... it works if you change it to this:

30.times { Kernel.sleep 0.05 }

I don't know why, but I'm assuming it has something to do with how sleep is implemented on Windows that causes it to lose the scope of self.interrupted. Or something. Whatever: this works.

Next, I'll try to find a regex that will actaully find the right files to run, when there has been a change! Grr.

Optics

2008-05-28T05:51:00.001-07:00

So, I was just thinking about microscopic images, and how they're always so much blurrier than illustrations. I was thinking about how, in videos, you can see various parts of the organism sharply, depending on the focus of the optics... it's just a razor-thin depth of field.

Then I thought: hey, wait, we have HDR photography. ...Why can't we use this at a microscopic level to increase clarity?

I could even imagine a machine-controlled microscope that produces one image from a composite of multiple images shot at different focal lengths, helping to bring the entire image into maximal focus.

...Of course, I'm sure someone else has already thought of this.

Posting Ruby

2008-05-21T15:47:00.001-07:00

So, I broke down and did some research on how to post code to blogger so that it doesn't look hideous. It wasn't nearly as simple as I wanted it to be (most notably, it was a serious pain to use cocoa to use the pasteboard for converting text)... so I hacked together a little rails project that handles things for me using web forms. It would be a long post to explain all the details, but I'll cover the basics.

First, sudo gem install syntax. Then create a new rails project and get it up and running. When you've got that, generate a controller called "Convert". The code for the controller should look like this:

class ConvertController < ApplicationController
  def ruby
    render :action => :ruby
  end
  def to_html
    require 'syntax/convertors/html'
    convertor = Syntax::Convertors::HTML.for_syntax params[:type]
    @code = convertor.convert(params[:text]).gsub(/<span class="punct">([^<]*)<([^<]*)</, "<span class=\"punct\">\\1&lt\\2<")
    render :action => :to_html
  end
end

And then you'll need a view for 'ruby.html.erb'.:

<h1>Convert What?</h1>
<p>
 <% form_tag('/convert/to_html') do %>
 <table><tr><td><%= text_area_tag 'text', '', {:cols => 80, :rows => 30} %></td>
   <td><%= radio_button_tag 'type', 'ruby', true%>Ruby<br/>
    <%= radio_button_tag 'type', 'xml'%>HTML/XML<br/>
    <%= submit_tag 'convert to HTML' ><td>tr><table>
 <% end %>
</p>

...And a basic response form in 'to_html.html.erb':

<h1>The raw HTML:</h1>
<pre><%= text_area_tag('whatever', @code, {:cols => 80, :rows => 30}) %></pre>
<h1>The results:</h1>
<%= @code.gsub(/&amp;/, '&') %>

...You'll also want to set up the CSS for your site, using the CSS from the site where I stole this idea from. Polishing things is an exercise left to the reader... but that should get you started.

Assumptions

2008-05-21T12:56:00.001-07:00

So, I was writing a few RSpec tests today, and required Hpricot to look at a few views. This was the code I wrote:

     links.each do |link|
        this_taxa_name = nil # scope
        img = link.at("img")
        if  img.nil?
          this_taxa_name = link.inner_text # This was the text link
        else
          has_image = true # This is the image.
          this_taxa_name = img['alt']
        end
        taxa_name ||= this_taxa_name
        assert_equal taxa_name, this_taxa_name, 'The names for the featured species were inconsistent (image alt/text)'
        taxa_link ||= link['href']
        assert_equal taxa_link, link['href'], 'The links for the featured species were inconsistent (image/text)'
      end
      assert has_image, 'There was no image for the featured species'

...This was all because I was trying NOT to assume that the image always precedes the text for the image. The test ended up being a bit longer than one page, so I asked a co-worker to look at it. His simple suggestion was "it's okay to assume the image comes before the text". By making that assumption, this block of code above becomes:

      assert !(links.nil? or links.length == 0), 'The featured species had no links'
      assert_not_nil links[0].at('img'), 'There was no image for the featured species'
      assert_equal links[0]['href'], links[1]['href'], 'The links for the featured species were inconsistent (image/text)'
      assert_equal links[0].at('img')['alt'], links[1].inner_text, 'The names for the featured species were inconsistent (image

It's very easy to see why assumptions are the #1 problem that developers have. They make life much, MUCH easier.

I suppose that's true of life as well.

Oh, and when adding methods to core classes...

2008-05-18T13:20:00.001-07:00

There's this neat pattern of checking to see if it's there first. I'm not sure if this really saves any cycles (I would imagine a few), but may aid in avoiding collisions, at the very least. For example thus:

unless String.instance_methods.include? "_sp_clean"
  class String
    def _sp_clean
      str = self.clone
      str = CGI.escape(str) # Turns spaces into '+' rather than '%20'... we expect that, though
                            # URI.escape allows [=&], etc... which we don't want.
    end
  end
end

...Now I hope that formats nicely when this message posts. : \

Built my first gem

2008-05-18T13:15:00.001-07:00

...And it was rather easy. After using the "newgem" gem to create the skeleton, I just followed some bloke's instructions for editing the Rakefile. Piece of cake.

Just thought I'd mention it.

Text and HTML in Thunderbird

2008-05-15T13:37:00.001-07:00

So, I'm using Thunderbird now.

One of the first pains that I've encountered is that, when replying to email, I get a warning that HTML may be unavailable to the recipient.

To turn this warning off (and select whatever you want as a default), look under Tools > Options > Composition > Send Options. The rest should be self-explanatory.

Learning

2008-05-15T11:22:00.001-07:00

Wow, what a blast these past three days have been! We had a consultant in from MySQL, and I learned hoards of new tricks, tips, and techniques. Here are a few:

You can store images on the filesystem based on their PK, to save having a field with the path. For example, image.id = 15579, image.path = images/1/5/5/7/9.png (or whatever level of separation you find appropriate).
EXT3--though it can be tuned--is not an ideal filesystem for databases. XFS is much better. (He didn't give specific reasons, and I don't particularly care to look into it.)
QCache_hits should be much higher than QCache_inserts. (show with "show global status like 'qc%'") If it's not, you need to hard-code which queries use the cache and which don't. Global status can also tell you about the number of locks that are occurring, which you may also wish to tweak.
query_cache_type values are: 0 = off, 1 = on, 2 = on demand
QCache hits are never logged. That would take more time. So if you want to really see how often queries are running, you have to turn of caches.
400M is a reasonable limit for query cache.
Stop apache from forking: this is what brings so many sites down. Instead, figure out how many concurrent processes your box can really handle, and then set the number of connections to this limit. If this is exceeded, the OS will queue additional requests, and keep the system from dying. 100 is a reasonable place to start.
MyISAM tables are fine if you're read-only (or nearly so): they will be faster. But InnoDB is much, much faster, since they store the table completely in memory. Of course, this means you have to have enough memory to fit the damned thing. ; ) ...And, of course, there are caveats about very large fields (like blobs, texts, and varchars with large average size). Shrinking your data as much as possible is important with large datasets... even one unused byte becomes a Meg with a million rows!
"show innodb status\g" is a tool for checking InnoDB performance.
strace shows all the system calls used for a command (linux). Very cool! Not sure how I hadn't seen this one before...
Sharding is what massive scaling is all about... but that's a topic you'll have to look up on the web, since there's so much to it. ; )

One more Annoyance with Tiger Terminal

2008-05-06T21:03:00.001-07:00

...So, when I moved .bashrc to .bash_profile, I lost my PATH environment variable. (Rather, it was stripped down to next to nothing.)

To fix this, I moved .bash_profile aside, re-ran terminal, ran env to get the old PATH, and pasted that into my .bash_profile where I used to say "$PATH". Something like this:

export PATH=/sw/bin:/sw/sbin:$GEM_HOME/bin:/opt/local/bin:/usr/local/bin:/opt/local/sbin:/usr/local/mysql/bin:/Users/cpr4cpu/bin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/X11R6/bin

...Pain in the arse, this.

Bash Profile, not BashRC

2008-05-06T20:36:00.001-07:00

Another lesson learned:

In Tiger, the Terminal doesn't used .bashrc. Why? Couldn't say.

Just move your .bashrc file to .bash_profile

Problem solved. Lame. There's more discussion about this on the Mac site, but I didn't really care to dig past the workaround.

gVIM

2008-05-06T20:29:00.001-07:00

I've spent the past couple of days setting up my Mac for Rails development. I'm still running Tiger, which is problematic (to say the least)... I hear that Leopard is all set up correctly, but I'm stuck for the time being.

I finally gave up on Aptana, which was just screwing up left and right. ...This means that I need to fall back to my old friend, Vim.

There's a script for calling gvim from a terminal, but I also wanted to change the color scheme on load (because my terminal is dark-on-light and my .vimrc colors are customized for that). To accomplish this, add to your .gvimrc:

colorscheme desert

(Or whatever other scheme you care for.)

This was surprisingly difficult to find (why?!?), so... I hope you stumbled on this page before too much extra work. ; )

Vim and Windows

2008-04-29T14:16:00.001-07:00

This is just a note for those who are searching the web...

So, today I was on someone else's machine, and noticed he used "File -> Open" from gVim... and I never do that! I just right-click and say "open with vim".

"Well," he explained, "when the file I'm opening is on the network, I don't get my _vimrc loaded".

This has never happened to me on my machine, so we looked into it. Turns out that gVim loads _vimrc from your %HOME% environment variable, but it does so from whatever drive your file is in. My %HOME% was specific about C:\, and his--actually just a reference to %HOMEPATH%--had no drive designation. When we added it, all was well.

The error message we were getting only came up when you quit gVim, and was something along the lines of "can't write _vimrc" or some-such.

Web Developers, PLEASE USE DATES

2008-04-29T07:05:00.001-07:00

If I could teach the world of web development one thing, it would be have a date on every relevant page.

Whether you like to admit it or not, someday your page is going to go stale, and the visitor needs some indication of this. I can't tell you how many times a month I'm at a site, find some interesting information, and later find out that it's totally useless because the page it came from is three years old.

Date everything.

...If you could teach webdevs one thing, what would it be?

Seeing Red (or Not)

2008-04-17T08:26:00.001-07:00

It's always interesting to have one's beliefs paddled over the head by science.

Today I learned that our blood is not blue, even when deoxygenated [via, via]. Who'd'a thunk?!?

In short, it turns out that our venous blood is actually deep maroon... ie, not much different from arterial blood. When vessels are close to the surface of our skin, we see the red. ...Which is why pale folk like me look vaguely pink. But as the vessels get deeper (to about .5 mm), some measure of absorbing and reflecting of light takes place, and you end up with a purple color. Thus, when compared to the pink of your skin, our eyes interpret that purple as blue.

Whacky! ...And humbling. There are still too many urban myths that I believe, even if I have become increasingly skeptical over the years.

Design Ideas from Nature

2008-04-14T19:54:00.001-07:00

Tonight I watched a series of videos on TED. One in particular got me thinking about how code should have more responsibility on a smaller level (the idea of the sticky foot doing everything it needs to on its own), and how failover might work (the idea of the leg acting as "foot" as needed).

Janine Benyus is not the most engaging public speaker, but it might be worth watching her speech as well, since there are some really huge ideas in there that could be applied to software. Lots of ideas of self-assembly... could data "flow" over code that captured the bits that belong there? Is this kind of what mapreduce is doing?

And this one is still one of my favorites... but that's more eye-candy than biomimicry:

Getting Ruby to Run on Tiger (as if you haven't heard that before)

2008-04-14T19:28:00.000-07:00

Today I had the [ahem] pleasure of getting ruby to run in Tiger.

I had been using Locomotive... but that appears to be terribly out-of-date (May 2007).

I had hopes that sakuzaku's script would do the trick... but it failed in several places. Afterwards, nothing worked. Honestly, I wasn't paying very close attention to it as it ran, and didn't crack it open to see what was going on... but that was my starting point.

I think the basic problem was that I had two other installations of ruby: one in /sw/bin/ruby, and the second in /usr/bin/ruby. ...So, after the script failed, I went and nuked those two executables and made them symlinks to /usr/local/bin/ruby.

But my gems were still miffed (I was trying to write a very, very simple script to go grab some XML from a web-app, and I was parsing it with hpricot). That turned out to be an easy fix:

export PATH=$GEM_HOME/bin:$PATH
export RUBYOPT=rubygems
export RUBYLIB=$GEM_HOME/lib/ruby/site_ruby/1.8/

(clearly, this is in bash) ...And that did the trick.

Yes, yes, I know: I need to upgrade to Leopard. But I'm running on a G5, and I've heard the performance hit is considerable... so I'm dragging my feet.