Clever::Tagline::Pending

Using cruisecontrol.rb with repositories without anonymous access

February 2nd, 2008 at 1:09 pm (Agile, Ruby)

I decided to use cruisecontrol.rb for continuous integration of an application that I'm working on at Bezurk. So I downloaded the 1.2.1 release from RubyForge and proceeded to add the project repository to the local installation of cruisecontrol.rb.

douglas@macbookpro:~$ ./cruise add MyProject --url http://path.to/repository --username 'douglas' --password 'guessable'
douglas@macbookpro:~$ ./cruise build MyProject
Builder for project 'MyProject' started
Logging to: /Users/douglas/Development/Ruby/cruisecontrolrb-1.2.1/log/MyProject_builder.log
Build loop failed
BuilderError: svn: PROPFIND request failed on '/svn/my_project/trunk'
./script/../config/../app/models/subversion.rb:98:in `execute_in_local_copy'
./script/../config/../lib/command_line.rb:86:in `call'
./script/../config/../lib/command_line.rb:86:in `e'
./script/../config/../lib/command_line.rb:84:in `popen'
./script/../config/../lib/command_line.rb:84:in `e'
./script/../config/../lib/command_line.rb:71:in `execute'
./script/../config/../lib/command_line.rb:70:in `chdir'
./script/../config/../lib/command_line.rb:70:in `execute'
./script/../config/../app/models/subversion.rb:89:in `execute_in_local_copy'
./script/../config/../app/models/subversion.rb:85:in `chdir'
./script/../config/../app/models/subversion.rb:85:in `execute_in_local_copy'
./script/../config/../app/models/subversion.rb:44:in `latest_revision'
./script/../config/../app/models/project.rb:228:in `new_revisions'
./script/../config/../app/models/change_in_source_control_trigger.rb:8:in `revisions_to_build'
./script/../config/../vendor/rails/actionpack/lib/../../activesupport/lib/active_support/core_ext/symbol.rb:10:in `__send__'
./script/../config/../vendor/rails/actionpack/lib/../../activesupport/lib/active_support/core_ext/symbol.rb:10:in `to_proc'
./script/../config/../app/models/project.rb:223:in `collect'
./script/../config/../app/models/project.rb:223:in `revisions_to_build'
./script/../config/../app/models/project.rb:202:in `build_if_necessary'
./script/../config/../app/models/polling_scheduler.rb:13:in `run'
./script/builder:79
./script/builder:78:in `catch'
./script/builder:78
./cruise:14:in `load'
./cruise:14:in `builder'
./cruise:68:in `send'
./cruise:68
/opt/local/lib/ruby/1.8/fileutils.rb:121:in `chdir'
/opt/local/lib/ruby/1.8/fileutils.rb:121:in `cd'
./cruise:67

Hmm, what's with the svn: PROPFIND error? Looking at the stracktrace doesn't tell me alot about what's going wrong here, let's try logging errors to the console.

--SNIP--
douglas$ svn --non-interactive info --xml
douglas$ svn --non-interactive log --revision HEAD:20 --verbose --xml
svn: PROPFIND request failed on '/repository/trunk'
svn: PROPFIND of '/repository/trunk': authorization failed (http://svnhost.com)
--SNIP--

It happens that my repository does not have anonymous access and requires a subversion user account to do anything useful. So it should be obvious that cruisecontrol.rb is trying to get log info from the repository but subversion is quitting with authentication errors because no user credentials are being supplied.

I need to have cruisecontrol.rb make use of the --username and --password options when making queries to the repository when I give it the credentials for access.

My first stop is the app/models/subversion.rb. Only the checkout method uses the username and password instance variables. Subversion should only include the --username and --password options when executing svn commands when both the username and password instance variables are present.

PLAIN TEXT
RUBY:




test/unit/subversion_test.rb


 


def test_svn_command_uses_user_password_when_provided


    svn = Subversion.new(:username => 'jer', :password => "crap")


 


    svn.expects(:info).with(dummy_project).returns(Subversion::Info.new(10, 10))


    svn.expects(:execute).with(["svn", "--non-interactive", "log", "--revision", "HEAD:10", "--verbose", "--xml",


                                "--username", "jer", "--password", "crap"],


                                {:stderr => './svn.err'}).yields(StringIO.new(LOG_ENTRY))


 


    svn.latest_revision(dummy_project)


end


 


app/models/subversion.rb


 


def checkout(target_directory, revision = nil, stdout = $stdout)


    @url or raise 'URL not specified'


 


    options = [@url, target_directory]


    options <<"--revision" <<revision_number(revision) if revision


 


    # need to read from command output, because otherwise tests break


    execute(svn('co', options)) do |io|


    begin


        while line = io.gets


            stdout.puts line


        end


        rescue EOFError


        end


    end


end


 


def svn(operation, *options)


    command = ["svn"]


    command <<"--non-interactive" unless @interactive


    command <<operation


    command += options.compact.flatten


    command += ['--username', @username, '--password', @password] if @username and @password


    command


end

The username and password would then be injected into the project's Subversion instance in the cruise_config.rb file for each project.

PLAIN TEXT
RUBY:




Project.configure do |project|


    project.source_control.username = 'douglas'


    project.source_control.password = 'guessable'


end

I've submitted a ticket along with a patch for this on cruisecontrol.rb's tracker. Keep a lookout for it if you happen to encounter the same problem.

2 Comments

Serializing custom Ruby classes with YAML

January 2nd, 2008 at 6:32 pm (Ruby)

I needed to be able to save an array of POROs (plain old Ruby objects) to the database on a Rails project that I'm currently working on. That should be easy, right?

This ought to do the job, yes?:class MyModel < ActiveRecord::Base serialize attribute_name end

However, this didn't quite work as advertised, the objects being returned were typed as YAML::Object instead of the actual class being serialized. Turns out that YAML is unable to find the reference to the actual class and so its falling back to using YAML::Object as the generic class type for deserialised objects. The answer to this: Rails ticket #7537.

Take a look at the YAML documentation for all the gory details.

Comments

Visualising Log Activity with Ruby and OpenGL with glTail

October 13th, 2007 at 1:05 pm (Ruby)

Here's an interesting combination of Ruby and OpenGL, glTail lets you visualise log activity in realtime. Check out the screencasts on the project page!

One thing to note at the time of writing, it assumes that the command for getting incoming log data, i.e "tail -f" is non blocking. So if you need to use "sudo tail -f", the application won't work properly for this situation. I might give it a shot at adding sudo support when I have some time to spare.

The quick spike I did for this suggests that I'll need to use a shell to respond to password challenges. Incidentally, Capistrano already has this figured out, so i'll be a good place to look for clues.

Comments

Raising URI::InvalidURIError from a perfectly valid URI

September 15th, 2007 at 12:47 pm (Ruby)

I was puzzled by URI::parse raising an URI::InvalidURIError on a perfectly well formed URI recently.

PLAIN TEXT

RUBY:

URI::InvalidURIError: bad URI(is not URI?): http://practicalguile.com/articles?query=latest
from /opt/local/lib/ruby/1.8/uri/common.rb:436:in `split'
from /opt/local/lib/ruby/1.8/uri/common.rb:485:in `parse'
from (irb):2
from :0

What's not apparent in this exception message is that the url contained a trailing space and this was causing URI.parse to fail. The following specifications demonstrate how it can trigger this particular exception.

uri.spec.rb

PLAIN TEXT

RUBY:

require 'rubygems'
require 'spec'
require 'uri'
describe URI do
it "should raise an InvalidURIException with leading whitespace in url" do
lambda{ URI.parse(' http://www.ruby-lang.org') }.should raise_error(URI::InvalidURIError)
end
it "should raise an InvalidURIException with trailing whitespace in url" do
lambda{ URI.parse('http://www.ruby-lang.org ') }.should raise_error(URI::InvalidURIError)
end
end

Running the spec will get you the result below.

ruby uri.spec.rb

..Finished in 0.030051 seconds

2 examples, 0 failures

Looking at the stacktrace in the exception, it's being raised by URI.split after URI.parse is invoked with the offending URL.

RUBY_INSTALL/1.8/uri/common.rb

PLAIN TEXT

RUBY:

def self.parse(uri)
scheme, userinfo, host, port,
registry, path, opaque, query, fragment = self.split(uri)
if scheme && @@schemes.include?(scheme.upcase)
@@schemes[scheme.upcase].new(scheme, userinfo, host, port,
registry, path, opaque, query,
fragment)
else
Generic.new(scheme, userinfo, host, port,
registry, path, opaque, query,
fragment)
end
end

Nothing weird happening in URI.parse, its a straightforward call to URI.split. So I'll go into URI.split, comments removed for brevity.

PLAIN TEXT

RUBY:

def self.split(uri)
case uri
when ''
when ABS_URI
scheme, opaque, userinfo, host, port,
registry, path, query, fragment = $~[1..-1]
if !scheme
raise InvalidURIError,
"bad URI(absolute but no scheme): #{uri}"
end
if !opaque && (!path && (!host && !registry))
raise InvalidURIError,
"bad URI(absolute but no path): #{uri}"
end
when REL_URI
scheme = nil
opaque = nil
userinfo, host, port, registry,
rel_segment, abs_path, query, fragment = $~[1..-1]
if rel_segment && abs_path
path = rel_segment + abs_path
elsif rel_segment
path = rel_segment
elsif abs_path
path = abs_path
end
else
raise InvalidURIError, "bad URI(is not URI?): #{uri}"
end
path = '' if !path && !opaque # (see RFC2396 Section 5.2)
ret = [
scheme,
userinfo, host, port, # X
registry, # X
path, # Y
opaque, # Y
query,
fragment
]
return ret
end

URI.split is matching the incoming url with an empty string as well as regular expressions for absolute and relative URIs. It's obvious from the specifications earlier that urls with leading/trailing whitespace do not match any of these and the case statement raises InvalidURIError, with the rather misleading message.

The regexes used for matching absolute and relative URIs is shown below, if you really want to know.

PLAIN TEXT

RUBY:

require 'uri'
include URI::REGEXP
ABS_URI
/^
([a-zA-Z][-+.a-zA-Z\d]*): (?# 1: scheme)
(?:
((?:[-_.!~*'()a-zA-Z\d;?:@&=+$,]|%[a-fA-F\d]{2})(?:[-_.!~*'()a-zA-Z\d;\/?:@&=+$,\[\]]|%[a-fA-F\d]{2})*) (?# 2: opaque)
|
(?:(?:
\/\/(?:
(?:(?:((?:[-_.!~*'()a-zA-Z\d;:&=+$,]|%[a-fA-F\d]{2})*)@)? (?# 3: userinfo)
(?:((?:(?:(?:[a-zA-Z\d](?:[-a-zA-Z\d]*[a-zA-Z\d])?)\.)*(?:[a-zA-Z](?:[-a-zA-Z\d]*[a-zA-Z\d])?)\.?|\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}|\[(?:(?:[a-fA-F\d]{1,4}:)*(?:[a-fA-F\d]{1,4}|\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})|(?:(?:[a-fA-F\d]{1,4}:)*[a-fA-F\d]{1,4})?::(?:(?:[a-fA-F\d]{1,4}:)*(?:[a-fA-F\d]{1,4}|\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}))?)\]))(?::(\d*))?))?(?# 4: host, 5: port) |
((?:[-_.!~*'()a-zA-Z\d$,;+@&=+]|%[a-fA-F\d]{2})+) (?# 6: registry)
)
|
(?!\/\/)) (?# XXX: '\/\/' is the mark for hostport)
(\/(?:[-_.!~*'()a-zA-Z\d:@&=+$,]|%[a-fA-F\d]{2})*(?:;(?:[-_.!~*'()a-zA-Z\d:@&=+$,]|%[a-fA-F\d]{2})*)*(?:\/(?:[-_.!~*'()a-zA-Z\d:@&=+$,]|%[a-fA-F\d]{2})*(?:;(?:[-_.!~*'()a-zA-Z\d:@&=+$,]|%[a-fA-F\d]{2})*)*)*)? (?# 7: path)
)(?:\?((?:[-_.!~*'()a-zA-Z\d;\/?:@&=+$,\[\]]|%[a-fA-F\d]{2})*))? (?# 8: query)
)
(?:\#((?:[-_.!~*'()a-zA-Z\d;\/?:@&=+$,\[\]]|%[a-fA-F\d]{2})*))? (?# 9: fragment)
$/xn
REL_URI
/^
(?:
(?:
\/\/
(?:
(?:((?:[-_.!~*'()a-zA-Z\d;:&=+$,]|%[a-fA-F\d]{2})*)@)? (?# 1: userinfo)
((?:(?:(?:[a-zA-Z\d](?:[-a-zA-Z\d]*[a-zA-Z\d])?)\.)*(?:[a-zA-Z](?:[-a-zA-Z\d]*[a-zA-Z\d])?)\.?|\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}|\[(?:(?:[a-fA-F\d]{1,4}:)*(?:[a-fA-F\d]{1,4}|\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})|(?:(?:[a-fA-F\d]{1,4}:)*[a-fA-F\d]{1,4})?::(?:(?:[a-fA-F\d]{1,4}:)*(?:[a-fA-F\d]{1,4}|\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}))?)\]))?(?::(\d*))? (?# 2: host, 3: port)
|
((?:[-_.!~*'()a-zA-Z\d$,;+@&=+]|%[a-fA-F\d]{2})+) (?# 4: registry)
)
)
|
((?:[-_.!~*'()a-zA-Z\d;@&=+$,]|%[a-fA-F\d]{2})+) (?# 5: rel_segment)
)?
(\/(?:[-_.!~*'()a-zA-Z\d:@&=+$,]|%[a-fA-F\d]{2})*(?:;(?:[-_.!~*'()a-zA-Z\d:@&=+$,]|%[a-fA-F\d]{2})*)*(?:\/(?:[-_.!~*'()a-zA-Z\d:@&=+$,]|%[a-fA-F\d]{2})*(?:;(?:[-_.!~*'()a-zA-Z\d:@&=+$,]|%[a-fA-F\d]{2})*)*)*)? (?# 6: abs_path)
(?:\?((?:[-_.!~*'()a-zA-Z\d;\/?:@&=+$,\[\]]|%[a-fA-F\d]{2})*))? (?# 7: query)
(?:\#((?:[-_.!~*'()a-zA-Z\d;\/?:@&=+$,\[\]]|%[a-fA-F\d]{2})*))? (?# 8: fragment)
$/xn

Looks rather intimidating, doesn't it? However, we're more interested in the beginning and end of the regular expressions so its safe to ignore all the stuff in between. Narrowing our focus down to the regex anchors (^ and $), we can see that there is no matching of whitespace, thus preventing a valid URI from being matched in URI.split.

This all means that URI.split has a undocumented pre-condition on the uri parameter being stripped of any whitespace around it.

2 Comments

« Previous Entries

Next Entries »

Clever::Tagline::Pending

Using cruisecontrol.rb with repositories without anonymous access

Serializing custom Ruby classes with YAML

Visualising Log Activity with Ruby and OpenGL with glTail

Raising URI::InvalidURIError from a perfectly valid URI

Pages

Listening to

Categories

Blogroll

Community

Currently Reading

Magazines

Recommended Reading