Baroquebobcat

This week I wanted to deal with one embarrassing bug, and some that looked fun. After looking through everything last week, I felt like I got a better sense about the current issues. So, I picked a few issues to work on that I thought I could dig into and get something working in an afternoon. I think I was a little ambitious.

First I wanted to fix #161 mirah -v causing an error because, frankly it’s embarrassing. I also thought it’d be pretty easy. It was, but the fix wasn’t as clean as I’d like.

Next time I’m in the option handling code, I’m going to do some house cleaning–it’s getting a bit ugly. For instance, parser should not take a list of commandline args and have to worry about ‘-e’ etc. How to reorganize it I haven’t decided. Maybe using optparse, though that has its own idiosyncrasies.

#13 booleans don’t have an == method.

I fixed this by adding an intrinsic to the jvm backend code. It’s not especially pretty. The intrinsics code is tied up in a few files and some of them feel like balls of mud. It’s not readily apparent where to find things and what they do.

I’ve been thinking about how to make intrinsics nicer and more consistent. It might be nice to have some common types with a common set of expected method definitions that the various (hypothetical) backends should implement. That way you’d have some consistency across different Mirah backends.

I’d also like to reorganize the intrinsics and make their internal APIs easier to grok. Things I’d like to do with them, like allowing easy method aliasing are hard right now, because the API has a lot of sharp edge cases.

Test cleanup

After fixing #13, I renamed all the test files. I was getting frustrated w/ having them all be prefixed with test_. That made tabbing into the right test file take a couple more keystrokes. It added just enough friction that I wanted to change it. So I did.

#30 Const Assign

This one is definitely not a single Sunday afternoon project. And, honestly I didn’t expect it to be. I spent about an hour trying to figure out what would need to change to support creating constants. It looks kind of annoying.

First, you need to transform the mmeta AST nodes into a Mirah AST node for the Const Assign, which could be as simple as creating a static FieldAssign. Then you need to change the code generation to deal with that. Unfortunately, FieldAssign’s don’t currently know about access levels (public, private, protected) which means you’d either have to add access levels to field assign, or create a new AST class that would have to be dealt with in the code generation phase.

Thinking about it a little, it might be nicer to do the second thing. Then, different (hypothetical) backends could handle access levels for constants their own way. That might be handy, particularly if constants are special in different ways in different languages.

#41 i++

The other thing I looked at briefly was adding ++ to the grammar. This turned out to be rather hard looking because the parser’s master branch is tied to mirah’s newast branch, which has a lot of new things in it and doesn’t work with the current release yet.

I’m debating creating a new branch on the parser at a point before it started using the new AST. On the other hand, it might make sense to spend more time trying to update the newast branch so that it is up to date with master. Then I could get it closer to merging back in.

That’s all for this week.

This week I decided to go and read all of Mirah’s open issues starting with the earliest submitted ones. I’d been spending so much time just looking at the top of the list. I didn’t have a good sense for which ones were duplicates, which were pretty undefined and which were easy. Thankfully, Mirah doesn’t have that many issues, so attempting to read through them all wasn’t a ridiculous undertaking. There were only 64, and I had filed about a half dozen of them myself.

In the end, I was able to close 7, and get a good sense of where a number of the other ones stood in terms of how much work it’ll take to fix them. Of course there were a few that made me confused, where I don’t have any idea where to start digging to fix them.

After going through the stack of issues, I have a better sense of what I want to try to tackle next week. Here’s a few that I think look fun.

#41 – adding ++ to the language.

++ isn’t in the grammar yet, so this would require learning more about the parser. And, I’d get to play around with the grammar, which is always fun, and something I haven’t done much of since graduating.

#127 & #42 – working out the semantics of equality.

Currently Mirah uses Java’s ==, which checks identity not equality. We’d like to use Java’s equals as our ==, but I’ve had a little trouble getting it working properly. Still, this is a fun one to hack on.

#45 – an issue with how field assignments check typing

Namely, they don’t really. It isn’t completely straight forward but I think I’ve nailed down where to make the changes.

#57 – who is self in a block

This looks fun. I’m not quite sure how to make it work. The problem is that you want blocks to work like they do in Ruby, where self is the owner of the outer scope. Currently, Mirah’s bindings don’t do that, so self is the instance of the closure. Looking at this got me thinking about non local return(NLR). There was an interesting discussion about it on Twitter today with @evanphx saying

I think it’d be really interesting to try to add NLR to Mirah’s blocks. It might be crazy, but it could also be awesome.

#69 – Mirah doesn’t check blocks signature against the signature of the method they’re supposed to be implementing.

Madness, but madness I think I can fix.

#74 Constants!

Mirah doesn’t let you create constants, other than classes and interfaces. I’d like to change that. And, the error tells you what’s missing. Now, all I’ve got to do is figure out how to implement it.

Tune in next week…

And that’s what I’m looking at doing next Sunday. Or some of it anyway. There’s a bunch of trickiness out there.

This week I looked at doing views in Shatner again. I had thought of a new approach to get around def_edb‘s issues with unquotes. That quickly devolved into an exploration of a part of the inference engine I hadn’t played with yet.

I’d been looking for a good way to share state between views and the ShatnerApp. Initially, my plan was to just have the view be a method on the Shatner app, just like how Dubious currently does it. But, that’s pretty pedestrian and totally wouldn’t cause things to blow up. Blowing things up in Mirah is one of the reasons I work on Shatner.

I thought maybe I could build view objects as closures. That way they’d have references to all the variables defined in the outer scope, but they could still be separate classes and have separate methods ala Rails’ view helpers.

And maybe it’d be easier to build a macro that just modifies the code in place to implement something like (gist):

so that it becomes (gist):

Of course, that didn’t work.

What I discovered was that Mirah currently doesn’t support closures being defined in closures (issue #155). They don’t work because they can’t figure out what the outer scope is.

When a closure is created in Mirah, it begins life as a Mirah::AST::Block. However, it doesn’t stay just a block for long. After the parse phase is over, it is transformed.

The typer figures out what method is being called with the block, and figures out what the block’s interface is. It generates AST nodes for a class that implements that interface and dumps the class into the tree of the outer class. It then adds AST nodes at the call site for creating a new instance of that freshly minted closure class and carries on typing.

The problem is that the AST nodes inside the block still think their parent is the block. This is only a problem when you have another block inside a block because a Mirah::AST::Block doesn’t have a class associated with it, so the closure generated inside the closure doesn’t know what contains it.

I still haven’t quite figured out how to resolve this. I feel like there’s probably some AST munging needed to get it to work properly, but I’m not sure what operations need to be taken.

It’s a very interesting problem though.

The other thing I looked at was getting == to use equals. Earlier, I’d tried copying String#+ doing something like (gist):

Which worked for ==, but not for !=, because I couldn’t figure out how to negate the call.

So, I tried the method used by kind_of?, which creates some AST nodes and shoves them back into the tree. That worked for the Java source generator, but not the Bytecode generator. Which was kind of odd.

Clearly I don’t know enough about how the bytecode generator works. Maybe I’ll look into that some next week.

‘Til next week everybody.

This week I decided to focus just on bugs. I picked a few that looked interesting and set about investigating.

I decided to look at #108, #150, #151, #68 and #52.

108 I realized, I hadn’t thought enough about. It’s really a design thing. Do you want to require that your closures match signatures with the methods the implement? What if you don’t care about the arguments because everything you want to do has to do with the closed scope and not with what you’re passed? I could see it both ways. On the one hand, requiring a matching signature ensures all the types and whatnot are correct. On the other hand, if you aren’t using the passed arguments, they are just boilerplate code.

I think we should allow closures to have no arguments where the method they are implementing has them for niceness. But it should be an all or nothing thing. Either no parameters, or parameters matching the signature of the method you expect to be implementing.

#52

I spent most of my time on this one. It seemed straight forward at first, but had a number of tricky bits that took some working out. The problem was that if you had a begin rescue end block, and treated it as an expression if the result types of the block itself and the rescue clauses differed, Mirah wouldn’t raise an exception, but the JVM would. For example (gist):

foo either is an int or a String, which would be fine in Ruby–but not in Mirah because Mirah has static typing.

So, I set about fixing it. The first thing was to define the valid behavior. What I came up with was this:

• if a begin rescue block is treated as an expression, check the compatibility of its result types
• if it’s not an expression, don’t check the result types

The second one is nicer because it lets you not care when you don’t want to care. For example (gist):

If we checked result types in both expressions and statements, the above would fail to compile, which would suck.

I started by writing a couple tests of the expected behavior. I wanted to get an inference error when I used the rescue as an expression, but not as a statement.

Then I went looking for other, similar pieces of functionality that I could use for a guide. I found that if elses also follow this pattern, so I wanted to look at how If AST nodes were inferred, trying to get a handle on how I would change rescues. I grepped through lib/ for If’s infer method and found it in lib/mirah/ast/flow.rb. What it does is get the if body’s result type, and the else’s result type and check to see if they are compatible (gist).

Rescue inference was a little more complicated than If because rescue’s have a lot more pieces. First, you have the body between the begin and the first rescue. Then, there can be multiple rescue clauses, each handling a different set of exceptions. Finally, if there’s an else, it runs after the body if there is no exception. There’s also ensure which is run after, but doesn’t return anything, so it shouldn’t affect result type comparisons (need to write a test for that …).

So, the types that need to be compatible are the that of the body if there isn’t an else or the else’s and all the rescue clauses. Kind of complicated. But pretty doable. I modified the Rescue node’s infer method to test compatibility of the body or the else and all the clauses, which worked for the tests I wrote but broke other tests.

It broke other tests because of how deferred inference works. When inference is reattempted, the typer assumes that the deferred node was evaluated as an expression, which breaks when it is a Rescue because its inference depends on how it was used.

So as a work around, I added an ivar to the Rescue node to cache how it was initially inferred (lib/mirah/flow.rb).

It’s a hack, but it resolved the issue. I’d like to look into how to change things to make expression vs statementness be something defined in the AST instead of decided at inference time, or make it more explicit that the expression-ness is determined by the parent node during inference.

This week I wanted to get back to my pet Sinatra clone–Shatner. Last time I tried working on it, I was trying to figure out how to get views working like they do in Ruby. I want to end up with a dsl like this(gist):

Where you can call edb :view_name just like you call erb :view_name in Sinatra, and it figures out what you want.

I tried to do this by injecting a call to the def_edb macro into the class’s AST node. That didn’t work. Because def_edb was getting expanded before the outer macro, instead of passing it a proper AST node representing a method name, I was passing it an Unquote, which it didn’t know what to do with.

This looked like a rather difficult thing to untangle, so I gave up on it for now and file a bug (#152).

I moved on to fixing some bugs.

I made a quick list of four boogs that looked promising

• #110 ICE with an empty block
• #108 closure generation
• #109 incorrectly attaching macros to the wrong node type
• #123 ICE from methods that have same name as previously defined macro

#110 ICE with empty block

Somewhere the AST node for the body of the block was being set to nil. This caused some later code to blow up when it attempted to iterate over the block’s children. I fixed it by ensuring that the body of a block is always set to some value.

I looked at some pretty interesting bits of the compiler while fixing this. Did you know that Mirah’s block support works in two ways? You can use it like you would in JRuby, and pass a block in place of an interface implementation, which is cool. And, you can also pass it a block that contains the definitions of the methods you want to implement. For example (gist)

#108 closure generation

I was able to reproduce this, and figure out roughly what’s going on, but haven’t fixed it yet. The problem is that we’re not generating the signature of the method a block passed as an interface correctly all the time. When the block has no arguments, but the method it’s supposed to implement does, Mirah generates a method with the same name but no arguments. Problem.

#109 incorrectly attaching macros to the wrong node type

I reproduced this but didn’t look into it. It’s interesting it only happens when the macro is used somewhere else in the code.

#123 ICE from methods that have same name as previously defined macro

The problem here was that a macro’s return type is InlineCode, and the MethodDefinition node didn’t know how to validate it’s own signature against a macro with the same name. I put in a condition that will cause inference to fail with an error that says the method has the same name as a macro. It’s not particularly elegant, but I don’t think it’s a bad thing for the compiler to do.

This week ended up mostly being researchy. I had a bunch of plans. I wanted to look at some of the work I’d done with implicit main methods–with an eye for changing from creating them at code generation time to modifying the AST and putting them in the right class. I also wanted to dig into making == use Java’s equals() instead of Java’s ==–which compares identity for objects.

Plans

• use AST modification for main method generation
• == –> .equals()
• eql? –> ==
• refactor intrinsic files
• research ivar inference(related #151)
• …
• boogs

I didn’t get to everything, but I touched a lot of it.

Main method

I had some old code that did this from the last time played around with it, so I tried starting there. It modified the JVM compiler’s define_main method to pull elements out of the AST and inject them into the main method of the class with the same name as the file. It worked as long as a class with a matching name was in the file and failed miserably when there wasn’t.

I tried different things to generate AST nodes for the ‘Script’ class that Mirah generates when there is no class of the same name as the file, but couldn’t get it to work. After taking a break and looking again I also realized that I was doing my editing in the wrong place.

The compiler is actually where the code generation happens and not where the AST is generated or transformed. It gets the AST after it has been transformed, as necessary, and after all the types have been inferred. Which makes it a bad place to try to extend Mirah’s behavior.

I think next time I’ll try to inject the main method AST transform as a plugin that adds functionality to the transformer.

equals

I poked around making == use Java’s equals method instead of using ==, which for objects compares their references to see if they point at the same thing. Getting == to work was actually pretty straight forward, I just replaced the code generation that does the java style == with a method call to equals. I didn’t push it yet because I haven’t added !=, which requires a little more knowledge about bytecode.

ivars

I wanted to see if I could figure out how to get public/protected instance variables from superclasses to be accessible from subclasses. I thought it would have something to do with look up mechanism, but I didn’t know where to start.

After looking around the various Java type lookup mechanisms I’m still a bit confused about how everything works. I think I’ll do a deeper look into it next week, focusing on this file (/lib/mirah/jvm/method_lookup.rb), which looks to be where most of the interesting type stuff happens.

Back at it, working on Mirah. This week I set my sights a little lower. Fix a few bugs, look more carefully at how different parts of the compiler works, you know the usual.

Plans

• Annotation bug (#148)
• == as equals
• …
• Profit

What I ended up doing was different, as usual.

#148

First, I looked into #148, because it looked interesting and probably touched parts of the compiler I wasn’t familiar with. The problem was that Mirah’s annotation support didn’t handle integers, so when you gave it one ala (gist):

It complained.

The problem was that neither code generator, bytecode or Java source, handled anything other than strings or arrays correctly. I fixed it naively, because I couldn’t come up with a better way–as I said it’s part of the compiler I’m not familiar with.

All I did was add a case for Fixnum, so other object types won’t work. I also changed the base case in the bytecode compiler so that it will raise an error with a note about what went wrong instead of something more obtuse.

I don’t think it would be hard to add additional cases–and maybe even generalize the way annotation values are handled, but I just wanted to fix this particular bug. And, do it in a way that would make it easy for someone to come and extend/improve it. That meant splitting out the test cases, which makes it easy to figure out where to put more tests for more annotation values.

Now, if you don’t really know Java, you might not know what an annotation is, in which case I suggest you skim the docs. It’s what I did.

One feature I’d like to figure out is how to add annotation creation support to Mirah. It’d be nice to be able to write annotations in Mirah.

JavaClass#java_method

Ran across this while trying to write a test case for #148. For future reference it takes as arguments.

• the method name
• argument types represented by java classes, ruby classes, or strings representing a java class(eg “java.lang.String”)

It took me a few tries to figure out how the argument types bit worked–I actually dug into jruby’s source a little to figure it out. For example (gist):

Handy test running snippet

When I run tests, I like to limit them by file, so I do it using Rake::TestTask‘s TEST= functionality. With TEST=, you can specify the file you want to run, and rake will only run that test file instead of the whole suite. It makes the feedback loop that much shorter which is really nice (gist).

#146

The other bug I fixed, #146, was pretty interesting. When I first looked at it, I thought it might have something to do with the macro that builds hashes from hash literals, which is something I’ve played with before. Which it did, but not in the way I initially thought.

The bug was that when you created a hash using literal syntax (gist):

when one of the values was created with a method call, it would fail to work with a weird low level Java problem (gist):

Further, what was weird was it DID work when you used static methods. I looked at the Java source generated by running mirahc -j hash.mirah, and there was a weird variable in there.

self$2000

Who was self$2000? What does it mean? Clearly it is not the self I was looking for. I guessed that the scope the hash was getting created in was being screwed up somewhere. The self it was attached to was being set wrong. Instead of being an instance of Foo, it was something else. Weird. So, I rolled up my sleeves and did a little spelunking (read: putting debug statements and binding.pry in places).

I found that hash literals are constructed using a macro that’s defined on the mirah.impl.Builtin. That in itself wasn’t terribly interesting, but what was interesting was that self was being set to mirah.impl.Builtin instead of Foo when the macro was being expanded.

So, I did what anyone would do trying to fix the problem at hand, I added a quick type check. The problem with the fix is that it doesn’t go far enough. Possibly, other classes that only contain macros could suffer from the same issue and this would not fix that.

Ideally, you’d have an annotation or something that you could use to tell when a class was used only for macros, and not reassign self in those cases. I can see a number of places where that could be really useful, eg extension classes that just contain macros acting on certain classes.

Well that’s it for this week. See you all next week.

After figuring out the class loading thing last week, we released a new version of Mirah(0.0.9) with the workaround so it’ll work on JRuby 1.6.4. Huzzah.

This Sunday, my head was a little groggy from facilitating a code retreat on Saturday (which went awesome). So, I ended up doing more code spelunking and less bug fixing. It’s cool though, because I had some interesting thoughts about some of the stuff I looked at.

Ye Olde Plans

• Fix the test_empty_array test.
• Look at boogs
• After parse callbacks
• ARGV/argv

Bonus

• refactoring???

Arr Boogs Ahoy

test_empty_array: So there was this pull request from a while back that I wanted to wrap up. Seems everybody else saw this test fail except me. Digging a bit, I figured out that there were two test cases used the same class name. Both of them had a test of that name. I changed the class name of one, and saw the error. Then, I merged the patch. Thanks thomaslee.

I spent a little time trying to figure out precisely what the test was supposed to be testing, but I couldn’t see a good way to make it more clear.

More boogs: class variables in class bodies(#113)

I looked at a bug about how class variables don’t seem to work in class bodies. There was a discussion about it on the mailing list last year, and I think we came to an agreement about it, but I don’t know if any of it has been implemented yet. As it is, variables in class bodies seem to do funny things, as you can see in the generated Java
in my comment.

Source Spelunking

From there I dove into the compiler’s source, looking for insight I guess. I wanted to know more about how things worked. I started by focused on looking at how inference works and how the AST and the typer interact. I have to say, I’m still a little confused about it, but I feel more comfortable than I did a year ago, when I first started looking at Mirah’s code.

Inference is done using a visitor pattern. A typer is passed through each node of the AST through the infer method, which takes a typer and a flag that says whether it’s inferring an expression or not. I still haven’t quite figured out what the flag does, but it seems important.

If the typer can’t figure out the type of a particular node on the initial pass, it defers figuring that node’s type til later. I think this allows you to do things like call methods that haven’t been seen by the typer yet.

So last week I got hung up on the classloader bug. This week I tried to find a workaround for it. It didn’t work but I learned a lot. Did too work!

Plans

• Bug Triage
• fix easy bugs
• class loader workaround
• after parse callback
• argv
• ….
• $$$$

I started with doing some bug triage. I looked at issues starting from the oldest in github issues, and tried to resolve them. I closed #26, #99, #119, #114. Some of them were already fixed, others were covered by other issues and some I fixed myself.

I also commented on a few issues asking for clarification.

#26 I couldn’t reproduce any more

#99 Isn’t exactly a bug, though it’d be nice if []= could be defined as a method without a macro.

#114 was a problem where a type error wasn’t being handled properly when a method was passed a block. The initial error caused the arguments lookup for the block to fail, blowing up.

#119 the problem here was that the method to transform an empty array literal wasn’t implemented. I cribbed the behavior of transform_zsuper to come up with something that worked.

After the bugs, I looked at

a class loader workaround

I thought, if the problem is that a class loader written in JRuby wouldn’t work, maybe one written in Mirah would. Unfortunately, I couldn’t get it working.

Converting the old code was surprisingly easy to do, which was nice. (gist’s below)

This:

became this:

But then I ran into a snag.

The problem was that the string containing the bytecode being passed around was getting converted to UTF-8. Which did not make Java very happy.

Java expects all it’s .class files to start with the value 0xCAFEBABE. Instead, the strings the class loader got began with 0xEFBFBDEF, which it didn’t like at all.

So, I went on a search for the right encoding. I tried passing UTF-8, UTF-16 BE and LE–the lot of the charsets on the Charset Javadoc–to getBytes in the ClassLoader. That didn’t work.

I tried changing the Map I was passing to the ClassLoader from classname -> String ‘o bytes to classname -> byte[]. That almost worked, but I couldn’t cast the byte array on the Mirah side.

Then I looked up binary encodings and Java and found this interesting gem. Turns out I’m not the only one whose run into this encoding issue. I followed the suggested workaround using the ISO-8859-1 charset. And it worked! Huzzah!

Unlike last week where I totally rocked it, this week I didn’t feel like I got as much forward motion. Too much time spent trying to diagnose #144 and not enough on other things.

What I set out to do
Boogs to smash

Features to rock

Experimenting

I like to be ambitious, what can I say.

Of that I got the class naming bug and one Java source bug fixed.

The rest of my time was spent on trying to figure out what was going on with the class loading problem. To be honest, I should have probably given up on it after I couldn’t figure out what was wrong and moved on to other bugs I could fix. Given more time, either I or somebody else would have come up with a solution. But, I love hard puzzles, and this was a good one.

The first thing I did was try to narrow down the possible places the bug was originating from. Unfortunately the stacktrace was pretty unhelpful in this regard because it didn’t tell me what happened between the outer call and the place where the class lookup failed.

I tried poking around and putting in debug statements. Eventually I decided that the bug was somewhere in the interpreter in something that had changed between 1.6.3 and 1.6.4.

Backup, what was the bug?
The bug was that when you attempted to load a macro, a ClassNotFoundException would be raised about the MacroClassName, with an extra ‘$’ on the end.

e.g.

java.lang.ClassNotFoundException: Square$Extension1$

So, I looked through JRuby’s source with an eye for the string “$”. I was figuring that where there was a $ appended to a classname, there might be some code related to the extra lookup that was failing.

rgrep ‘\”$\”‘ src

Mostly I found false positives, there are a quite a few places that deal with “$”s. But not too many that append “$”‘s to strings. I found a couple things that stood out, and one in particular was especially interesting because it added a “$” to the end of a class name and then attempted to load it. And, the method it was in was changed between 1.6.3 & 1.6.4. handleScalaSingletons

To be honest, that wasn’t really the point I knew that it was the problem. I did a whole bunch more debugging, including trying out git’s bisect feature (which is pretty awesome, by the way).

handleScalaSingleton previously caught Exception, but in 1.6.4 it was changed to catch the specific exceptions that can be raised from loading a class. The problem was that the exception being raised was wrapped.

JRuby internally wraps exceptions raised in Ruby in a RaisedException class, which is fine in itself. It unwraps it before sending back to client Java methods, when it makes sense. The problem it that it is passed around in the interpreter as a RaisedException, not as the type of native exception that was raise from Ruby. When a class loader implemented in Ruby raised a ClassNotFoundException, it would get wrapped and raised in the interpreter as a RaisedException, so it wouldn’t get caught by a catch for a ClassNotFoundException.

There’s now a pull request for fixing it that @abscondment wrote.