Monday, 4 February 2008

Parentheses in the calculator: a first look at context-free grammars

This series on writing a program that can evaluate simple arithmetic expressions is all about baby steps. I have no idea what I'm doing in terms of the problem space and I'm similarly ignorant of my tool, Ruby. But I'm making progress and have a good idea of what is next:

  • expand number support to the rationals
  • report the position of a syntax error when found
  • implement support for parentheses
The first two shouldn't be a big deal, I could probably get them working pretty quickly with the existing code base. Parentheses however, are significantly more work. I know of only one way to deal with them: context-free grammars. Even if I was aware of another way to deal with the parens I would ignore it - this exercise is about learning compilers and I know that implementing a context-free grammar is necessary. I have found the lecture notes for CS 164 at Berkeley University to be quite useful. They read well without a lecturer and handle the topics with a degree of patience that allows me to comprehend it. I really wish I had been given the chance to do a course like this while I was at university. I think the context-free grammar I need can be defined as follows:
E => n | E+E | E-E | E*E | E/E | (E)
E stands for Expression and is the only non-terminal in this grammar. A non-terminal is a token that can be "expanded" further using the grammar rules. The grammar contains the following terminal tokens: 'n', '+', '-', '*', '/', '(' and ')'. Basically this grammar says that an expression can consist of either a number (n), or a combination of two expressions, or an expression in parentheses. Using this grammar we could derive the following strings:
120+16*2 6+((120-2)*4) (((1)))
But note that we couldn't derive:
1+*2 )2+3 4+(-)
Obviously this is the sort of behaviour we want. So now that I have a grammar, I want to implement a function that will take a string and tell me whether it is derivable from the grammar. There are a few ways to do this but I have to read more before I can figure out which one to try to implement first. Its also possible that I will have to modify my grammar to make it parse able.

Saturday, 2 February 2008

The calculator, version one

There are a few loose ends to tie up and then the first pass of the calculator is complete. If you've forgotten where I was up to, here is a quick reminder:

  • The calculator falls over when the numbers get big because I implemented the tree evaluation in a educational but less than practical manner.
  • Whitespace should be ignored.
  • The codebase needs a general cleanup.
None of these are particularly involved - this should be a relatively short post! During the last post I discussed a few ways to implement the tree evaluation. I pointed out the most obvious solution (a switch statement), mentioned polymorphism and then ventured forth into using unbound methods. Upon discovering the issues with the latter approach it was a no brainer to go back to polymorphism:
class OperatorToken
 attr_reader :multitive, :value

 def evaluate(container_node)
   return do_operation(container_node.left_child.evaluate(), container_node.right_child.evaluate())
 end 
end

class AdditionToken < OperatorToken
 def initialize()
   @value = "+"
   @multitive = false
 end

 def do_operation(left_child_value, right_child_value)
   return left_child_value + right_child_value
 end 
end
I think you can guess what the other Tokens look like. The @value field isn't necessary to the implementation but it allowed me to keep a bunch of existing tests. The @multitive field is a replacement of the crappy named "major" field I used to determine whether the operator was higher precedence. I had to make a few minor changes to other sections of the program but it was all very easy. Whitespace is trivial to deal with. I could have easily handled it when implementing the tokenization but I guess it slipped my mind. Lets add a test for it:
  def test_spaces
   assert_equal([14,"+",62,"*",127], tokenize_for_value("  14 +62*   127 ") ); end
The fix is just stripping whitespace from the string each time a token is generated. I added a few small tests and did my best to clean up the code. You can download it here. Some sample output:
C:\Users\Paul>irb
irb(main):001:0> require 'calculator_v1'
=> true
irb(main):002:0> calculate "10 + 2*6"
=> 22
irb(main):003:0> calculate "1000000000000000000000000000000000-1"
=> 999999999999999999999999999999999
irb(main):004:0> calculate "x-1"
=> "Syntax Error"
irb(main):005:0>
Next post I discuss the what I would like to achieve in the next version.

Friday, 1 February 2008

Thoughts on Android Code Day London

I spent today at Google London participating in an Android Code Day. The structure of the event was pretty simple. Roughly, it was:

  • 2 1/2 hours: presentation and questions.
  • 2 1/2 hours: "hacking".
  • 1 hour: networking (no, the OTHER type of networking, you know, the one that involves humans communicating - not machines).
I don't intend to report on the details of the day - but I have some observations. I attended the event in a non-professional capacity and in that regard I was definitely in the minority. Many of the attendees were there to assess the platform, rather than code against it. Many of the questions boiled down to "how are you going to stop the manufacturer/network operator from screwing this up for everyone by doing X?". I think working in the mobile development industry must make you jaded. Our host, Jason Chen, was great. He had an excellent blend of technical and presentation skills and also a great deal of patience. I suppose its given for a developer advocate to have these traits but there was one other thing I liked about Jason: he was genuine. A developer advocate is supposed to be enthusiastic about the product - its a fundamental part of their job! But sometimes this can be taken too far and results in a credibility drain. With Jason this was certainly not the case. Unfortunately, there was not enough Jason to go around. For an event advertised with a focus on coding, I was disappointed that there wasn't more Google staff attending to assist developers. Somehow I managed to break the debugger bridge between Eclipse and the Android emulator on my laptop and it wasted a lot of my time. It would have been nice if there was a group I could join that was discussing how the integration between Eclipse and the emulator works, perhaps it would have helped me solve my problem. Instead the groups that formed were oriented on different types of apps: locational, multimedia based, etc. I'm sure this was great for some, perhaps the majority - it just didn't work out so great for me. On a related note, I think the general lack of focus for the coding portion of the day was a problem. There are probably lots of different ways you can run an effective session aimed at developers but simply letting them loose for 2.5 hours isn't one of them. A live tutorial that you can code along with would have been cool. Perhaps something that picked up where the online notepad tutorial left off. Another idea would have been to prepare some sort of incomplete application the developers could download over the wifi and amend. Couple it with a list of small features that the developers can attempt to implement within the session and I think you have a much more compelling activity. But hey, this is the first time I've been to a "code day" so what do I know? I don't really fit the target profile. It was fun to see some of the Google offices. They are quite colourful. I enjoyed the opportunity to speak briefly with a couple of Googlers, especially Jason. Although I had a few negative things to say about the event in this post, I still enjoyed it and am glad I went. You can be sure that there will be plenty of Android related posts appearing here in the coming months but before that happens I will be returning to my parsing project. I have to be careful about letting myself get spread across too many different things at once. This is another good example of how maintaining this blog is beneficial - it greatly increases my chances of finishing the projects I start.

Monday, 28 January 2008

Evaluating the expression tree

Lets start off with some tests:

class EvaluateTests < Test::Unit::TestCase
  def test_1
    assert_equal(1, calculate("1"))
  end
  
  def test_add
    assert_equal(3, calculate("1+2"))
  end
  
  def test_subtract
    assert_equal(3, calculate("7-4"))
  end
  
  def test_multiply
    assert_equal(6, calculate("3*2"))
  end
  
  def test_divide
    assert_equal(3, calculate("6/2"))
  end
  
  def test_negative_result
    assert_equal(-1, calculate("1-2"))
  end
  
  def test_multitive_before_additive
    assert_equal(14, calculate("4*3+2"))
    assert_equal(14, calculate("2+3*4"))
    assert_equal(10, calculate("4+3*2"))
  end
  
  def test_left_to_right
    assert_equal(8, calculate("12-6+2"))
    assert_equal(4, calculate("12/6*2"))
  end
end
Not exactly comprehensive, but its a start. I need to write a calculate method to tie together the work I've done over the last few weeks:
def calculate(expression)
  tokens = tokenize(expression)
  root_node = build_tree(tokens)
  return root_node.evaluate()
end
Evaluating the tree is obviously a recursive function. Here is the first bit:
class BinaryTreeNode
  def evaluate()
    return @token.evaluate(self)
  end
end
I decided the easiest way to do this is have the evaluate method take an argument: the tree node that contains the token. When evaluating an IntegerToken the value is simply returned, no need to use the argument:
class IntegerToken < Token
  def evaluate(containerNode)
    return @value
  end
end
For the OperatorToken, we want: evaluate(left child) [operator] evaluate (right child). There are lots of different ways to do this - a case statement is the simplest:
class OperatorToken
  def evaluate(containerNode)
    case @value
      when "+"
        return containerNode.lhs.evaluate() + containerNode.rhs.evaluate()
      when "-"
        return containerNode.lhs.evaluate() - containerNode.rhs.evaluate()
      when "*"
        return containerNode.lhs.evaluate() * containerNode.rhs.evaluate()
      when "/"
        return containerNode.lhs.evaluate() / containerNode.rhs.evaluate()
      end
    end
   end
end
My my isn't that boring. Another option would be to implement subclasses for OperatorToken - I wouldn't learn much from doing it though. Lets try something different instead:
class OperatorToken < Token
  attr_reader :major
  
  def initialize(value)
    @value = value
    @eval_method = Fixnum.instance_method(value)
    case @value
      when "+","-"   
        @major = false
      when "*","/"
        @major = true
      end
  end
  
  def evaluate(containerNode)
    bound_method = @eval_method.bind(containerNode.lhs.evaluate())    
    return bound_method.call(containerNode.rhs.evaluate())
  end
end
The three relevant lines of code are highlighted. But first, some background. Operators in Ruby are actually just methods. When I execute the line x = 2 + 3 I am calling the '+' method on the object '2' which is an instance of the class Fixnum. So what I am actually doing with that line is this: x = 2.+(3) . Classes in Ruby are just like normal objects, so you can call methods on them as I am doing in the first highlighted line. If the OperatorToken is constructed with "+", then I acquire an instance of the '+' method defined on the class Fixnum and store it as @eval_method. What I actually store is an instance of UnboundMethod - if you are a .NET developer, it might help to think of it as similar to MethodInfo as you can use it to invoke the method, but you will need to specify the instance you wish to invoke the method on. The second highlighted line does this - the left node is evaluated and the result is our bind target. The third line invokes the bound method and passes the result of evaluating the right node. The result is what we are after and is returned. Yes, this is much more complicated than a switch statement, and surely much slower. But I learned something, and hopefully so did you. Lets run my unit tests:
>ruby calculator.rb
Loaded suite calculator
Started
........................
Finished in 0.022 seconds.

24 tests, 37 assertions, 0 failures, 0 errors
>Exit code: 0
I think its worth mentioning that I have sort of covered up some mistakes I made while implementing the tree building. I didn't initially write the left to right unit tests and so for a while my calculator could do "2+9/3*4" but not "9/3*4" (it returned zero rather than 12). I didn't notice this problem until I ran the stress tester:
def stress_test()
  expressions = generate_expressions(100)
  rejects = []
  expressions.each do |exp|
      rejects << exp if (eval(exp) != calculate(exp))    
  end
  return rejects
end
 
def generate_expressions(count)
    expressions = []
    operators = ["+","-","*","/"]    
    count.times do
      exp = (rand(100) + 1).to_s
      rand(10).times do
        exp << operators[rand(4)]
        exp << (rand(100) + 1).to_s
      end        
      expressions << exp
    end
    return expressions
end
This baby will generate 100 expressions, each containing up to 10 terms between 1 and 100. It does not generate zeros because I would then have to deal with division by zero problems (I'll let that wait for version two). The eval method is not mine - it is a built in method that uses Ruby's own expression evaluation engine. This is really useful because I can pass an expression string into my program and compare the result with what Ruby came up with. If there is a discrepancy then I add it to the array of rejects and return them when done. The stress test revealed the aforementioned problem with left to right evaluation which I fixed, but it also raised something else. Consider this test:
def test_really_big_number
  assert_equal(1000000000000000000001, calculate("1000000000000000000000+1"))
end
Unfortunately, it doesn't pass:
>ruby calculator.rb
Loaded suite calculator
Started
........E................
Finished in 0.02 seconds.

1) Error:
test_really_big_number(EvaluateTests):
TypeError: bind argument must be an instance of Fixnum
calculator.rb:55:in `bind'
calculator.rb:55:in `evaluate'
calculator.rb:12:in `evaluate'
calculator.rb:118:in `calculate'
calculator.rb:209:in `test_really_big_number'

25 tests, 37 assertions, 0 failures, 1 errors
>Exit code: 1
When a Fixnum gets larger than a certain size it changes into a Bignum. Unfortunately, this wreaks havoc with my evaluation code because my unbound method will only bind to a Fixnum, not a Bignum. There are some workarounds, a relatively simple one already springs to mind. However I don't think they are worth entertaining - going back to the case statement or something similar seems to be a better idea. Next post I clean up this issue and the code in general.

Pasting code into Blogger

I've been using a javascript tool to format my example code. It does two useful things:

  1. Replaces html-unfriendly characters such as <>
  2. Wraps it in some nice formatting tags.
You can choose whether it embeds the style information directly into the output or creates a seperate css entry. I've been using the embedding up until recently, but now I have a problem. I've discovered that Blogger's rich text compose mode annihilates my indenting so its time to switch to using the html editor instead. The embedded styles make for noisy html so I will fiddle with this post and try to get the css working instead.
class EvaluateTests < Test::Unit::TestCase
  def test_1
    assert_equal(1, calculate("1"))
  end
end
Edit: Yay, that works fine.

Sunday, 27 January 2008

More syntactic analysis: building an expression tree

Its been a few days since the last post in this series, mostly because this step was trickier than the previous few and partly because once I had the tree building working I couldn't resist adding the evaluation step. This was actually a good thing, because it helped me discover a mistake I made with the tree building and saved me from making a fool of myself by posting some broken crap (writing unit tests should have theoretically prevented this but I suck at them). Anyway, lets get on with it. I need to define the data structure the tree will be built from. I tried enhancing the tokens to act as the tree nodes themselves, but it made the tree manipulations more complicated. Instead I decide to keep it simple:

class BinaryTreeNode
attr_accessor :lhs, :rhs, :token

def initialize(token)
@token = token
end

def leaf?
return @lhs == nil && @rhs == nil
end
end
Pretty self explanatory I think. The tree node holds a reference to a token and its two child nodes. The 'leaf?' method isn't really necessary but it helps simplify the building algorithm. I considered trying to download some sort of diagramming tool to explain the process, but I knew it was going to suck. So instead, you get this: Hopefully you can follow this without too much trouble. I build the tree by performing the same single operation over and over. I don't know what to call it ('create_parent' isn't particular explanatory), but this is the code:
def create_parent(target_node, operator, integer)
new_node = BinaryTreeNode.new(operator)
new_node.lhs = target_node
new_node.rhs = BinaryTreeNode.new(integer)
return new_node
end
This method takes an existing node and makes it the left child of a new node. It then sets up the operator and the value and returns the new node. In the diagram above, this method is called on the (6) node in step 1 and the result is the tree in step 2. The tree building algorithm only needs to use this operation on two nodes:
  • the root node if the operator is a + or - or if the root node is the only node or if the root operator is a * or /.
  • the root node's right hand child in other cases (the operator is a * or / and the other conditions don't apply).
Once you understand the create_parent operation, you will find there is not much else to it:
def build_tree(tokens)
tokens = tokens.dup
root_node = BinaryTreeNode.new(tokens.shift) 

while(tokens.size > 1)
  operator = tokens.shift
  integer = tokens.shift 
  if(root_node.leaf? || !operator.major || root_node.token.major)
    # If the root_node is the only node or if we have a '+' or '-'
    # OR if the root operator is '*' or '/' then append to root
    root_node = create_parent(root_node, operator, integer)   
  else
    # We have a * or /, append to root.rhs   
    root_node.rhs = create_parent(root_node.rhs, operator, integer)            
  end
end

 return root_node
end
The Operator.major method is a badly named way of determining whether the operator is addition or subtraction (minor) or multiplication or division (major). I'm sure there is some appropriately mathy term that I could use instead but it did not come to me. So while there are still tokens left to process, I take two off, and then do a create_parent with them. The logic I described above is used to figure out where the new node is created and thats it. I must admit I was pleased with the simplicity of this approach. If the next operator is addition or subtraction then you want the existing expression to be evaluated first, so simply creating a new root node and attaching the existing tree to the left does the job. If the next operator is multiplication or division, then you want it to take precedence before the root operator, UNLESS the root operator is also multiplication or division (to preserve left to right evaluation). You might be wondering what that first line of code is, 'tokens = tokens.dup'. This was a quick fix for a small issue I overlooked. I am using Array.shift to work my way through the tokens and as a result when the calling code returns, the array it passed in will be empty. The 'dup' method simply returns a clone of the array which I can modify without risk of breaking anything. I tried writing unit tests for the tree building but they end up being quite lame. Its fiddly work to verify via code that the nodes are arranged into the correct structure. I know it goes against the spirit of unit testing but I found it easier to test the tree structure simply by skipping forward to implementing evaluation and using the evaluation tests. Tomorrow I will post the wacky implementation I chose for evaluating the tree. Its probably not the best way to do it but it demonstrates some cool Ruby functionality.

Thursday, 24 January 2008

Android Code Day: London

In November last year, Google announced an open source operating system for mobile phones called Android. This is a big deal for me because I have wanted a "gPhone" for a while now and a phone that runs Android will be the closest I can get any time soon. I expect the mobile applications market will grow significantly over the next few years and so this is one of the fields I want to start spending time on. I was expecting to start posting about Android in a week or two, after I finished the series of articles on the parsing/calculator exercise. The reason why I'm posting about Android today is because last night I managed to register for the Android Code Day running next week at Google London. I found out about it via one of Google's official blogs and I'm quite glad I was subscribed - I checked a few minutes ago and registration has already closed. These things are popular! I'm still working on the parsing exercise. I can be kinda slow when it comes to this sort of stuff (hey that's part of why I'm doing it!) and I'm struggling to come up with the algorithm for building the binary tree while ensuring I follow order of operations. I could probably Google/Wiki it but its become a personal challenge now - I know I can do this bit without help, its just about getting my brain to shift into the right gear. "You are SUCH a geek – you know that don’t you?" - My boss's reply when I asked for the day off so I could go to the Android Code Day :)