v0.6 Release – Fixing Annotate for the Paper Theme

Posted on February 17, 2009 by Sid

In my last release I had put out an hg annotate fix for the gitweb_mozilla theme. Dirkjan Ochtman, a Mercurial Project developer, noticed my release and asked me to come up with a similar fix for the Mercurial Project’s paper theme. I decided to take up the task and see if I could get similar results for the paper theme as I did with gitweb_mozilla.

The paper theme uses a HUGE table to display the results just like gitweb_mozilla. I tested the current version of the theme using a large 10,000 line cpp file. It gave me a loading time of ~30sec, which is ~10sec longer than gitweb_mozilla.

The following is the code to fix annotate:

map

annotateline = '<div class="l#parity#"><div class="codeauthor"><a href="{url}annotate/{node|short}/{file|urlescape}{sessionvars%urlparameter}#{targetline}" title="{node|short}: {desc|escape|firstline}">{author|user}@{rev}</a></div><a class="codeline" href="#{lineid}" id="{lineid}">{linenumber}</a>{line|escape}</div>'

style-paper.css

div.codeauthor { 
    display:inline-block; 
	width:16ch; 
	text-align: right; 
	color:#999999; 
	text-decoration:none;
	margin-right: 25em;
} 
a.codeline { 
    color:#999999; 
	text-decoration:none; 
	margin:0 10px; 
}                                   
div.l0 {
    background-color:#f6f6f0;
}                                                                    
div.l0, div.l1 { 
    display:block; 
}
pre.completecodeline { 
    font-size: 90%;
	line-height:1.4em; 
	font-family: monospace;
    white-space: pre;
}
div.headrev {
    float: left;
	margin-right: 32em;
	margin-left: 8em;
	font-size: 90%;
	font-weight: bolder;
}
div.headline {
	font-weight: bolder;
	font-size: 90%;
}

I changed the code (see above) and tested annotate again to examine the difference in loading time. I got some surprising results. The reduction in file size was not significant at all. The fix for gitweb_mozilla brought down the file size by 25%. However, in this case the reduction in file size wasn’t nearly as significant. Also the reduction in loading time for gitweb_mozilla was ~15sec but for the paper theme the loading time was only reduced by ~12secs. Currently, on my machine the loading time has gone down from ~30secs to ~18secs.

The speed increase isn’t as signficant for the paper theme as it was for gitweb_mozilla. Why is that? I don’t exactly know. Obviously there are other factors coming into play that aren’t allowing a similar speed boost. Nonetheless, there is a noticeble increase in loading times.

Posted in DPS911, Mercurial Project, Open Source | Tagged , , , , | Leave a comment

v0.6 Release – Minor fix for bug 445560

Posted on February 16, 2009 by Sid

All the way back in November 2008 I had put out a patch to implement expand and collapse functionality (for merge changesets) for the pushlog. I had gotten an r+ review but I just needed to make some minor adjustments to the patch. Time went on and I totally forgot about implementing that minor fix. I was looking through some of my previous work and then I realized that I had forgotten about this.

The change is pretty simple but important. I just have to use proper naming conventions for variable names, which I wasn’t doing before. This might be the type of thing that is easily overlooked, I was guilty of that myself but it is important that we abide by naming conventions. Why? So that the code is readable when other people inevitably come along to change/read it.

In hgpoller/pushlogfeed.py I changed “Id” to “id” as all other variable names are non-capitalized:

 entry = {"author": ctx.user(),
       "desc": ctx.description(),
       "files": web.listfilediffs(tmpl, ctx.files(), n),
       "rev": ctx.rev(),
       "node": hex(n),
       "tags": nodetagsdict(web.repo, n),
       "branches": nodebranchdict(web.repo, ctx),
       "inbranch": nodeinbranch(web.repo, ctx),
       "hidden": "",
       "push": [],
       "mergerollup": [],
       "id": id
       }

In hg_templates/pushlog.tmpl I changed id to pushid to make it more clearer:

var pushid = $(this).attr("class");
pushid = '.' + pushid.substring(11, pushid.length);
$(pushid).nextAll(pushid).toggle();
Posted in DPS911, Mercurial Project, Open Source | Tagged , , , , , | Leave a comment

v0.6 Release Goals

Posted on February 15, 2009 by Sid

It’s time to start working on my 3rd release for this semester. My goal is to put out 3 patches this time around:

These 3 patches should combine to make a good solid release. These are 3 seperate improvements to hgweb that I’m sure users must be looking forward to having. I’m not quite sure what changes have been made to pushlogfeed.py causing it to bitrot. The solution to the problem may be simple or it may be complicated. I was under the impression that no changes needed to be made but new code has been added causing my code to break. Another exciting prospect is that I’ll have my name added to the Mercurial Project once I implement the annotate fix for the paper theme making my work available to all hgweb users.

I’ll be putting all the details of my work on the project page and this blog. Time to get to work!

Posted in DPS911, Mercurial Project, Open Source | Tagged , , | Leave a comment

v0.5 Release Complete

Posted on February 7, 2009 by Sid

I’ve finally completed my 0.5 release. This release I tackled an interesting problem with hg annotate. Trying to improve efficiency and loading time of an application isn’t something I had tackled before. It was a good experience trying to figure out the solution to this problem. I’ll be putting up a new patch on the bug page very soon. The following are some of the important links for this release:

View the project page for more details.

EDIT: I’ve posted the patch for this release. Have a look…

Posted in DPS911, Mercurial Project, Open Source | Tagged , , | Leave a comment

v0.5 Release – Running an Experiment

Posted on February 7, 2009 by Sid

In my last post I revealed my fix for the hg annotate loading issues. My fix reduced the loading time to a relatively reasonable ~8sec considering the fact that currently the loading time is ~20sec. However, it was still bugging me that Mat’s implementation was producing faster loading times than my fix. His implementation is ~1sec faster.

Mat’s Implementation

The problem is that this implementation doesn’t use valid HTML. Mat uses two non-standard tags, x and l#parity#. Although, I must say that this is a very unorthodox and smart solution to this problem:

annotateline = '<l#parity#><x><a href="#url#diff/#node|short#/#file|urlescape#{sessionvars%urlparameter}">#author|obfuscate#@#rev#</a></x><a href="##lineid#" id="#lineid#">#linenumber#</a>#line|escape#</l#parity#>'

Altering Mat’s Code into Valid HTML

As an experiment I decided to test what would happen if I took Mat’s code and replaced the x and l#parity# tags with divs. Would the implemenation still remain as fast? Would it still be faster than mine? The following is the altered code:

annotateline = '<div class="l#parity#"><div class="codeauthor"><a href="#url#diff/#node|short#/#file|urlescape#{sessionvars%urlparameter}">#author|obfuscate#@#rev#</a></div><a class="codeline" href="##lineid#">#linenumber#</a>#line|escape#</div>'
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

div.codeauthor { 
    display:inline-block; 
    width:16ch; 
    text-align: right; 
    color:#999999; 
    text-decoration:none;
    margin-right: 25em;
} 
a.codeline { 
    color:#999999; 
    text-decoration:none; 
    margin:0 10px; 
}                                   
div.l0 {
    background-color:#f6f6f0;
}                                                                    
div.l0, div.l1 { 
    display:block; 
}
pre.completecodeline { 
    font-size:12px; 
    line-height:1.4em; 
}
a.codeline:hover,
a.codeline:visited,
a.codeline:active {
    color: #880000;
}

Results

I tried testing the altered code and I was getting a loading time of ~7sec for this cpp file. However, I was never able to get into the ~5sec region which Mat’s original code was sometimes able to achieve. This altered version was only able to reach a minimum time of ~6sec.

The point is that this altered version is a bit faster than my fix but slower than Mat’s original implementation. I compared the file sizes of the 3 versions:

  • My fix: 2.4MB
  • Mat’s Fix:1.8MB
  • Altered version of Mat’s Fix: 2.2MB

It is interesting to note that using div tags instead of an x and l#parity# tag increases the file size by 18%. I don’t know why that happens but somehow the x and l#parity# tags are more efficient than div tags. Nonetheless the reasons don’t matter, the altered version of Mat’s fix seems to be the best solution to this problem at this time.

Posted in DPS911, Mercurial Project, Open Source | Tagged , , , | 2 Comments

v0.5 Release – My Fix for the hg Annotate Problem

Posted on February 6, 2009 by Sid

In my last post I had examined a couple of patches that other people had posted for bug 459823. This time I wrote my own code to see if I could make an improvement on what others have done. I did borrow some things but my implementation will be a bit different.

HTML

Instead of using one large table to hold all the rows I’ve decided to use a single div that holds two links and another div:

annotateline = '<a class="codelineauthor" href="#url#diff/#node|short#/#file|urlescape#{sessionvars%urlparameter}">#author|obfuscate#@#rev#</a><a class="codelinenumber" href="##lineid#" id="#lineid#">#linenumber#</a><div class="codeline parity#parity#">#line|escape#</div>'

The HTML is pretty simple, nothing fancy. All the formatting will be done by the CSS instead of a table.

CSS

I’ve decided to place my CSS in the style-gitweb.css style sheet instead of just leaving it in the fileannotate.tmpl file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

a.codelineauthor {
	width: 50ch;
	float: left; 
}
a.codelinenumber:hover,
a.codelinenumber:visited,
a.codelinenumber:active {
    color: #880000;
}
a.codelinenumber {
    text-decoration: none;
	margin-right: 0.5em;
	float: left;
	color:#999999;
}
div.completecodeline {
    font-size: 12px;
	line-height: 1.4em;
	font-family: monospace;
	white-space: pre;
	display: block;
}

The float property is used to align the two links and the div so that it looks like a table.

Result

The result is quite positive. This implementation cuts the loading time by ~12sec when testing a 10,000 line cpp file in mozilla-central. The loading time currently on live is ~20sec and my implementation cuts it to ~8sec. This is around the same amount of time that Bonsai takes to load the same file. The browser doesn’t freeze and the initial screenful appears immediately just as jorendorff specified.

It is interesting to note that the size of the page on live (with the HUGE table) is 3.2MB while my solution cuts the file size down to 2.4MB, a 25% reduction.

Posted in DPS911, Mercurial Project, Open Source | Tagged , , , | Leave a comment

v0.5 Release – Examining Two Patches Regarding the Hg Annotate Problem

Posted on February 4, 2009 by Sid

In my previous blog post I had a look at the default and coal templates for hgweb in order to find a fix to the hg annotate problem. Neither of those templates presented a viable solution so the next step I decided to take was to have a look at a couple of patches that people have attached to the bug, which apparently reduced the loading time by a bit.

Patch by Smaug

Smaug is currently the person assigned to the bug and he put up a patch back in 2008 that got an r+ rating from ted. However, it doesn’t do exactly what jorendorff wants. The patch displays a loading message until the entire page is done loading and then shows the code. Instead, jorendorff wants the first screenful to appear almost right away. Thus, I decided to remove the loading message and then test the loading time of annotate. I was able to get a time of ~12sec to ~14sec on my machine when loading this file.

Obviously this is an improvement in loading time (currently the loading time is ~20sec). However, I want it to be even faster. One great thing about this patch is that now you can select code without selecting the line numbers. Another important change is that the huge table is now gone, replaced by div and pre tags. There are some things here that I can definitely use. I will be using div tags in my implementation as well but I don’t know if I will be doing it the same way as Smaug.

Patch by Mats Palmgren

The next patch I examined also gets rid of the table and uses a pre tag to display the code. I tested it on my machine and got a loading time of ~7sec to ~8sec. Now, this is a definite improvement over the current loading time. The increase in speed over the current implementation is very noticeable. However, the problem with this patch is that it doesn’t use valid HTML. I want my implementation to use valid HTML so I can’t really use the same type of implementation as this patch. I’ll have to come up with a more conventional approach.

Looking Ahead

I think by now I’ve researched all the resources that I could acquire. Now, I can get down to actually writing the patch. I’m pretty sure I know what to do, lets see how it turns out. Stay tuned!

Posted in DPS911, Mercurial Project, Open Source | Tagged , , | Leave a comment

v0.5 Release – Examining How Other Templates Implement Hg Annotate

Posted on February 3, 2009 by Sid

In order to complete the goals for my v0.5 release I examined how other templates implement hg annotate. I wanted to find out whether these other implementations had the ability to load 10,000 line cpp files quickly or not.

Default Template

Hgweb comes with a default template, which is quite ugly if you ask me but maybe it is efficient. I put together my local hgweb environment to use the default template and I was able to get a time of ~12secs. However, there was a major problem with this template. I noticed that after about 2000 lines the rest of the lines stopped showing up. The page kept loading new content but as I scrolled down the rest of the page was completely blank. None of the other 8000 or so lines of code were visible. This problem means that this template is of no use to me.

Coal Template

The next step was to try the coal template, which strangely doesn’t look very different from the default template. I configured my local hgweb environment to use the coal template and I got a time of ~14secs. This is not a significant improvement over Mozilla’s template which was clocking in at ~20secs but it is definitely better. Any reduction in loading time is a good thing. The problem is that gitweb_mozilla (the template that Mozilla uses) and the Coal template both use the same type of implementation. They both use one HUGE table to display the file, which obviously is not the correct method to employ (see below).

gitweb_mozilla

<div class="page_body">
  <table border="0">
    #annotate%annotateline#</table>
</div>

coal

<div class="overflow">
  <table class="bigtable" border="0">
    <tr>
      <th class="annotate">rev</th>
      <th class="lineno">line</th>
      <th class="line">source</th>
    </tr>
    {annotate%annotateline}
  </table>
</div>

Neither of these templates have been able to help me to solve my problem. Next I’ll be having a look at some of the patches that people put up on the bug page. Lets see if they improve annotate’s loading time or not.

Posted in DPS911, Mercurial Project, Open Source | Tagged , , , | Leave a comment

v0.5 Release – Understanding the Problem with hg Annotate

Posted on February 2, 2009 by Sid

The last couple of days I’ve been examining the problem with hg annotate in order to complete the goals for my v0.5 release. The issue is with the template Mozilla is using for hgweb. The problem has already been worked by other people but nobody seems to have come up with a viable solution.

Research

I decided to start by doing some research. I read the discussion on both the Mozilla and Mercurial bugs. Then I used Google to look for howtos on reducing page load times. I actually was able to find a couple of good references that every web developer should read (view here and here). I also tried some of the patches that attempt to solve the problem posted on the bug page. Also I looked into how fast hg annotate operates with the default and coal templates. I only saw a significant increase in speed with one of the patches.

Testing

I ran hg annotate on my local machine and I am getting an average time of approximately 20secs for the page to fully load. I measured this using firebug, which is an incredible tool by the way.

Looking Ahead

The problem is staring me right in the face. I need to find out why it is occuring and then remedy it somehow. Currently the page loads a HUGE table containing thousands of rows. This obviously has to go but just getting rid of that doesn’t cut down the load time by any significant amount. I’ll have to examine the CSS and see what kind of changes I can make to improve efficiency. There are some recommendations in the bug discussions, which I will definately try out. Lets see what I can do…

Posted in Mercurial Project | Leave a comment

v0.5 Release Goals

Posted on January 28, 2009 by Sid

My v0.4 release is over and done with. In that release I mainly focused on bug 459727 but for my v0.5 release I’ll be taking a look at a problem with hg annotate, bug 459823. The problem here is that hg annotate takes FOREVER to load large files such as ten thousand line cpp files found in mozilla-central. People have already spent a bit of time working on this bug so this gives me some insight intto what I’m required to do.

I had a chat with jorendorff and we discussed what should happen when using hg annotate:

  • Show the first screenful in a second or less
  • Don’t lock up the browser while the rest loads
  • Don’t take forever to load the rest of the file, because I often want to search for a snippet of code

This is an interesting task for me as I’ve never really looked at how firefox renders things on the page. In order to make hg annotate load things faster I’ll have to look at what is more efficient. For example would a page with a table of 10,000+ rows load faster or a page with 10,000+ divs? Right now, I don’t know but I’ll have to find out because as jorendorff put it “the current code just fails all day long” and something must be done about that.

Posted in DPS911, Mercurial Project, Open Source | Tagged , , , | Leave a comment