Posts Tagged pushlog

v0.7Release - Retrieving the Total Number of Entries

Earlier today I outlined the goals of my current release. One of the problems I was looking to fix was with the function, getMaxEntries(). The job of this function was to create an xmlHttpRequest() in order to retrieve the maximum number of push entries in the repository. The pushlog displays data in reverse chronological order and thus I need to know the maximum number of entries so that I know which entries to display first to maintain the same order.

For example in my test repository I have 2613 total entries. The first 10 entries are displayed by default so when the user scrolls down to load more data the very first entry shown would be #2603, then 2602 and so on. Previously, to retrieve the maximum number of entries I was using the following function:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
function getMaxEntries() {
  var entries = new XMLHttpRequest();
  var max = 0;
  entries.open('GET', '/json-pushes?startID=0&endID=1', true);
  entries.onreadystatechange = function() {
  if(entries.readyState == 4)  {
    if(entries.status != 404) {
      var entryData = JSON.parse(entries.responseText);
      max = entryData[1].max;
 
      start = max - 10;
    }
  } else 
      return 0;
  } 
  entries.send(null);
}

I didn’t really like this method of using an xmlHttpRequest() just to retrieve one value. I had to call this function onPageLoad() to calculate the maximum number of entries. This had to be done before my OnScroll function was called or various horrible errors would occur.

I’ve noticed something very unique about JavaScript. It doesn’t wait for one function to finish executing before going on to the next function. If your first function is taking too long, JavaScript will move on to your next function. Now, this feature was causing me quite a few headaches. As I said before, getMaxEntries() had to finish executing completely before the OnScroll function was called, otherwise things would go horribly wrong. getMaxEntries() sets the value of start, which is then used by loadEntries() to retrieve data (called OnScroll). Since getMaxEntries() was taking too long to execute, JavaScript was moving on to loadEntries() without setting the value of start, completely wrecking my logic.

Also, getMaxEntries() was causing Firefox to freeze for 10-15secs for reasons unknown to me. This way of calculating the max number of entries wasn’t very elegant and totally unacceptable. I needed to come up with a better solution.

The Solution

hgweb uses a template system called genshi. Basically, the maximum number of entries is calculated via a database query on the server side. In order to pass this server side variable to the client side I had to add the line 4 to pushloghtml() in hgpoller\pushlog-feed.py:

1
2
3
4
5
6
7
8
9
10
    return tmpl('pushlog',
                changenav=changenav(),
                rev=0,
                max=query.totalentries,
                entries=lambda **x: changelist(limit=0,**x),
                latestentry=lambda **x: changelist(limit=1,**x),
                startdate='startdate' in req.form and req.form['startdate'][0] or '1 week ago',
                enddate='enddate' in req.form and req.form['enddate'][0] or 'now',
                querydescription=query.description(),
                archives=web.archivelist("tip"))

Now, this change ^ allowed me access to this variable on the client side. All I had to do was to use this format: {<var_name>} or #<var_name>#. However, one draw back as far as I know is that I can only access this varialble in HTML, not in JavaScript, which is a significant drawback.

So in order to retrieve this data I added an id attribute the following div tag in hg_templates\gitweb_mozilla\pushlog.tmpl:

1
<div id="#max#" class="page_header">

Now, I could set the value of start to the maximum number of entries:

1
start = $("div").attr("id");

Thus, I can easily pass the correct value of start to loadEntries(), which makes an xmlHttpRequest() to retrieve the correctly ordered data when the user scrolls down.

, , , , , ,

No Comments

v0.4 Release Goals

The 0.4 release deadline is approaching fast (next week) and I thought it would be beneficial to come up with a list of goals that I will be attempting to complete. I will be applying further fixes to bug 445560. So, to get to the point straight away the following are my goals:

Optimize the client side code

The goal is to maintain/add functionality but decrease the lines of code. Understanding the current code I have come up with can be confusing. I want to make it more efficient and slim it down a bit. I think I can do this by moving some functionality to functions as some lines of code are being repeated.

Fix the buglink feature

The buglink feature works but there are some bugs strings that it doesn’t pick up. It won’t pick up strings with the patterns “123456″ or “b=123456″. Also if there is more than 1 potential bug link in the string only the very first one is converted into an actual bug link. Case in point below:

Fix the onPageLoad feature

The pushlog should load enough changeset entries onPageLoad so that the vertical scroll bar appears. Now, the problem is that different people use different screen sizes so just loading a static number of items onPageLoad is not enough. The users screen size must be taken into account. Currently, the number of entries are loaded according to the users screen size but I’m getting an unresponsive script error (see below). It does work if the user clicks continue. I want to get rid of this error so that the functionality works smoothly. One foreseeable problem is that I will have a hard time testing this feature since I only have a 15 inch screen.

You can follow my progress on the project’s wiki page.

, , ,

No Comments

Another Patch Approved, for bug 445560

I’m happy to report that another patch of mine has been approved by ted! This is for bug 445560. For this patch I had to make some changes after I initially put up the patch but they weren’t major ones. For more details about this bug and what my patch does visit my project page.

After my first patch that was successful, this is the second patch that I have gotten approved from the work I have done throughout the semester. Hopefully pushlog users will be utilizing this feature very soon! It just gives me a sense of delight when something I have worked hard on gets approved so that it is available to the users. I feel like doing a little dance right now!

, ,

No Comments

v0.3 Release - Fixing the Problems with my Patch for bug 459727

I’ve been busy trying to fix problems that had arised regarding the patch I submitted for bug 459727. All that is being done right now is fixing various small problems but the overall functionality remains the same. So the following are the solutions to the problems that jorendorff identified:

To calculate the max SELECT COUNT(*) FROM … should be used

Changed my implementation to use gettotalpushlogentries(conn) which performs a select statement. I’ve gotten rid of getMaxEntries() since it is no longer required.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
def pushes_worker(repo, startID=0, endID=None):
    stmt = 'SELECT id, user, date, rev, node from pushlog INNER JOIN changesets ON id = pushid WHERE id > ? %s ORDER BY id ASC, rev ASC'
 
    args = (startID,)
    if endID is not None:
        stmt = stmt % 'and id <= ?'
        args = (startID, endID)
    else:
        stmt = stmt % ""
    if os.path.basename(repo.path) != '.hg':
        repo.path = os.path.join(repo.path, '.hg')
    conn = sqlite.connect(os.path.join(repo.path, 'pushlog2.db'))
    pushes = {}
    for id, user, date, rev, node in conn.execute(stmt, args):
        mergeData = []
        ctx = repo.changectx(node)
        if len(ctx.parents()) > 1:
          for cs in ctx.parents():
            mergeData.append(hex(cs.node()) + '|-|' + clean(person(cs.user())) + '|-|' + clean(cs.description()))
        if id in pushes:
            pushes[id]['changesets'].append(node)
            pushes[id]['mergeData'].append(mergeData)
        else:
            pushes[id] = {'user': user,
                          'date': date,
                          'changesets': [node],
                          'formattedDate': util.datestr(localdate(date)),
                          'individualChangeset': hex(ctx.node()),
                          'author': clean(person(ctx.user())),
                          'desc': clean(ctx.description()),
                          'mergeData': mergeData,
                          'max': gettotalpushlogentries(conn)
                          }
    return pushes

`new Function(”return ” + entries.responseText) ()` is saying
`eval(entries.responseText)`, perhaps use `JSON.parse (entries.responseText)`

I haven’t implemented this yet because I want more information on why the way I have done is not the appropriate way. I’ve asked on the bug page (comment 12)but haven’t heard back yet

The page loads more results when the user scrolls.  What if the initial results don’t fill up the window?

I added the following CSS to always show a scrollbar no matter what. This should allow the user to scroll even if the initial entries don’t fill up the page.

html { 
  overflow: scroll; overflow-x: auto; 
}

getMaxEntriesis called, then ’start’ is used once the user scrolls.  But the user might scroll before the first result comes back, while start is still zero — a race condition

I changed the implementation to check the value of the start variable before more entries are allowed to load onScroll. If the value of start is still 0 then no entries are allowed to load. The following is the code:

$(window).scroll(function() {
  if(start > 0) {
    if($(window).scrollTop() == $(document).height() - $(window).height()) {
      renderMorePushLogResults();
    }
  }
});

Better to move CSS and formatting to stylesheets where possible

I removed explicitly setting the styles of the rows. Now I am just setting the class name of the row so that it uses the preexisting styles for that class name. View lines 37-43 for the code

The JS code isn’t consistently indented

It should be correctly formatted now. View here

This patch seems to contain some non-ASCII characters.  Bugzilla doesn’t render it properly

Used &mdash in the places where non-ASCII characters were being used. View lines 73 and 121

In a few places the b tag is used where the server uses the strong tag

Changed the b tags to use strong. View lines 73 and 121

I will be putting up a new patch very soon…

EDIT:

The new patches have been posted here

, , , , , , , ,

No Comments

My patch for Bug 448707 Approved!

Some good news to report today. My patch for bug 448707 has been approved by jorendorff. He did find some problems but they were minor enough that they could be ignored for now.

Getting an r+ on the first go is pleasently surprising. Hopefully I can keep moving along and get a patch up for the graph view very soon.

, , ,

No Comments

v0.2 Release Getting Merge Changesets to Display Properly for bug 459727

I’ve been working hard to make merge changesets display properly OnScrollDown for bug 459727. They weren’t appearing at all before since I didn’t implement any code for them. I started to work on doing so but I ran into a puzzling problem that jorendorff helped solve. I’ve managed to get merge changesets to display properly now. It wasn’t as hard as I had expected but the amount of code I had to write was quite long. There are probably ways to shrink it down a bit but I just don’t see how right now. Lets go over it…

The Code

The server side code can is explained here. The following is the client side code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
function renderMorePushLogResults() {
  //start += 20;
  //end = start + 19;
  var end = start;
  start = start - 20
  var pushCheckins = new XMLHttpRequest();
  pushCheckins.open('GET', '/json-pushes?startID=' + start + '&endID=' + end, true);
  pushCheckins.onreadystatechange = function() {
  if(pushCheckins.readyState == 4)  {
    if(pushCheckins.status != 404) {
      var loader = document.getElementById("loader");
      loader.innerHTML = '<img src="{url}static/ajax-loader.gif" align="right" />';
 
      var pushData = new Function("return " + pushCheckins.responseText) ();
      var counter = 0;
      for(var i = end; i > start; i--) {
        var trScroll = document.createElement("tr");
 
        if(counter == 0) {
          counter = 1;
        } else {
            trScroll.style.backgroundColor = "#f6f6f0";
            counter = 0;
        }      
 
        var tdScrollUser = document.createElement("td");
        tdScrollUser.width = "184px";
        tdScrollUser.innerHTML += '<i>' + pushData[i].user + '<br />' + pushData[i].formattedDate + '</i>';
 
        //Create changset link
        var tdScrollChangeset = document.createElement("td");
        tdScrollChangeset.innerHTML += 
         '<a href=\"/rev/' + 
         pushData[i].individualChangeset.substring(0, 12) + 
         '\">' + 
         pushData[i].individualChangeset.substring(0, 12) + 
         '</a>';
 
        //Create buglink
        var bugInDesc = (pushData[i].desc).toLowerCase().indexOf("bug");
 
        if(bugInDesc != -1) {
          var bugLinkName = (pushData[i].desc).substring(bugInDesc, bugInDesc + 10);
          var bugNumber = bugLinkName.substring(4, 10);
          var bugLink = 
           (pushData[i].desc).substring(0, bugInDesc) + 
           '<a href=\"https://bugzilla.mozilla.org/show_bug.cgi?id=' + 
           bugNumber + 
           '\">' + 
           bugLinkName + 
           '</a>' + 
           (pushData[i].desc).substring(bugInDesc + 10, (pushData[i].desc).length);
        } else { //No bug provided
            var bugLink = pushData[i].desc;
        }
        var tdScrollAuthorDesc = document.createElement("td");
        tdScrollAuthorDesc.innerHTML += '<b>' + pushData[i].author + ' — ' + bugLink + '</b>';
 
        trScroll.appendChild(tdScrollUser);
        trScroll.appendChild(tdScrollChangeset);
        trScroll.appendChild(tdScrollAuthorDesc);
 
        loader.innerHTML = "";
        document.getElementById("titlePush").appendChild(trScroll);
 
        //Check whether it is a merge changeset or not
        if(pushData[i].MergeData != []) {
          for(var j = 0; j < pushData[i].mergeData.length; j++) {
            if(pushData[i].mergeData[j] != "") {
              var mergeStr = pushData[i].mergeData[j];
              for(var k = 0; k < pushData[i].mergeData[j].length; k++) {
                var actualMergeStr = mergeStr[k].split('|-|');
 
                var mergeC = actualMergeStr[0];
                var mergeUser = actualMergeStr[1];
                var mergeDesc = actualMergeStr[2];
 
                if(mergeDesc != pushData[i].desc) {
 
                  var trScrollMerge = document.createElement("tr");
                  trScrollMerge.style.backgroundColor = trScroll.style.backgroundColor;
 
                  var tdScroll_MergeUser = document.createElement("td");
                  tdScroll_MergeUser.width = "184px";
 
                  //Create changset link
                  var tdScroll_MergeC = document.createElement("td");
                  tdScroll_MergeC.innerHTML +=
                   '<a href=\"/rev/' +
                   mergeC.substring(0, 12) + 
                   '\">' + 
                   mergeC.substring(0, 12) +
                   '</a>';    
 
                  //Create buglink
                  var merge_bugInDesc = mergeDesc.toLowerCase().indexOf("bug");
                  if(merge_bugInDesc != -1) {
                    var merge_bugLinkName = mergeDesc.substring(merge_bugInDesc, merge_bugInDesc + 10);
                    var merge_bugNumber = merge_bugLinkName.substring(4, 10);
                    var merge_bugLink =
                    mergeDesc.substring(0, merge_bugInDesc) + 
                     '<a href=\"https://bugzilla.mozilla.org/show_bug.cgi?id=' + 
                     merge_bugNumber + 
                     '\">' + 
                     merge_bugLinkName + 
                     '</a>' + 
                     mergeDesc.substring(merge_bugInDesc + 10, mergeDesc.length);
                  } else { //No bug provided
                      var merge_bugLink = mergeDesc;
                  }
                  var tdScroll_MergeAuthorDesc = document.createElement("td");
                  tdScroll_MergeAuthorDesc.innerHTML += '<b>' + mergeUser + ' — ' + merge_bugLink + '</b>';
 
                  trScrollMerge.appendChild(tdScroll_MergeUser);
                  trScrollMerge.appendChild(tdScroll_MergeC);
                  trScrollMerge.appendChild(tdScroll_MergeAuthorDesc);
 
                  document.getElementById("titlePush").appendChild(trScrollMerge);
                }
              }
            }
          }
        }
 
      }    
    } 
   }
  }
  pushCheckins.send(null);
}

Everything before line 68 is for normal changesets that don’t have any merge changesets (in which case the whole section encapsulated within the if statement is skipped). Everything afterwords sets up the merge changesets.

Line 69 is a for loop that goes through pushData[i].mergeData which contains all the data for the merge changesets (changeset, user and description). Then on line 73 a split() is performed (the data is received as a string in the format: changeset|-|user|-|description) to get the values of the changeset, user and description. Instead of putting the data within a character delimited string I could have created a class but this implementation meant writing less lines of code which is almost always better.

On line 81 the row is created and the background color is set. The first column is then created (it will be empty) and the width is set to get the right alignment. Lines 87-95 the changeset link is created. Lines 97-114 the buglink is created via some cumbersome string manipulation. Lines 116-118 all three tds are added to the row and then on line 120 the row is added to the table.

The Result

The above code gives me the following result: http://sidkalra.com/files/mercurial/mergeChangsets.png

, , , , ,

No Comments

v0.2 Release - Concerning the Chronological Order for bug 459727

Currently the pushlog loads entries in reverse chronological order (the latest entries are shown on top instead of on the bottom). I want to maintain this functionality for the new entries that are loaded OnScroll. For my current implementation this is not occurring. Have a look at this example: http://sidkalra.com/files/mercurial/improvedPushlog2.png

Examine the date of the last entry before new items are loaded (Mon Sept 29 17:20:28 2008 - 0400) and then have a look at the date of the first new entry (Fri May 02 14:38:33 2008 - 0400). Even though this is reverse chronological order it is not correct since all the entries between May 02 and Sept 29 are missing.

Currently to load more entries OnScroll I am using a script that returns all the data I need. This script takes in two values, startID and endID. So this is what I’m doing currently:

  1. On first scroll: startID = 0, endID = 20
  2. On second scroll: startID = 21, endID = 40
  3. On third scroll: startID = 41, endID = 60
  4. and so on…

This obviously isn’t getting me the right data. I don’t understand which startID and endID I should start with. Having the startID = 0 obviously isn’t working. I’m kind of stuck on this issue.

From what I understand, the whole point of this feature is to mirror the functionality of pressing the Next link. When a user scrolls to the bottom of the page, the new entries should be identical to the entries that appear if one were to click on the next link.

, , , , ,

No Comments

v0.2 Release - Adding a loader GIF to Pushloghtml’s OnScroll feature

These days it seems like every AJAX application has some sort of loader GIF to let the user know that new data is being loaded. It has almost become a standard and lets one know that they are using an AJAX application. I wanted to add this functionality to pushloghtml’s OnScroll feature, bug 459727

So, I went hunting for some a good loader GIF that I could use and in the process I found this nifty little site: http://www.ajaxload.info/, which lets one create a custom loader GIF.

The next step was to decide where to place the actual GIF on the server so that I could call it in the pushlog.tmpl file. This had me stumped for a while. I tried placing the GIF in various places but nothing seemed to work. Then, I noticed there were pre-existing images on the pushloghtml page and they must be stored somewhere on the server so I did a little search for *.jpeg and found the location of the images.

To show the GIF before new data is loaded I placed the following code in my OnScroll function:

1
2
var loader = document.getElementById("loader");
      loader.innerHTML = '<img src="{url}static/ajax-loader.gif" align="right" />';

Then, right before the very first row is displayed on the screen with appendChild() I remove the GIF by doing the following:

1
2
loader.innerHTML = "";
        document.getElementById("titlePush").appendChild(trScroll);

The above code shows the GIF OnScroll (bottom-right corner): http://sidkalra.com/files/mercurial/loader.png

, , , ,

No Comments

Looking Forward to v0.2 Release

The date for my v0.2 Release is approaching rapidly and it’s time that I decide on what exactly I will be working on. After a good bit of thought I have decided that my best option is to work on the following:

  • Continue working on bug 459727 to make pushloghtml show more entries onScroll
  • Add a fix to ted’s patch for bug 445560 so that when the user clicks on a particular [Expand] link it only expands the changesets belonging to it and not all changesets on the page.
  • Add functionality to show which files were touched by a changeset and roughly the size of the diffs. Maybe add onMouseOver functionality to show it in an unobtrusive way.

I think adding these 3 features will be more than enough for v0.2 release in terms of work load. I have already spent countless hours working on the  first 2 features. Also I think these 3 features will benefit the community the most by improving the core features of the pushlog. I’ve consulted with jorendorff on this and he seems to agree that these 3 features would be good enough for my next release.

, , , , , , ,

No Comments

Getting pushloghtml Up and Running Part Two

Carrying on from where I left off , my problems with python modules continue. After some conversations with djc, bsmedberg, ted and jorendorff it seems that Mercurial is using an entirely different python interpreter and that is why even though my 2 modules are installed properly hg serve is still throwing me an error.

I installed my Mercurial using a windows binary and it seems that it is based on py2exe, which zips up its own libraries and installs them separately along with a separate Python interpreter. Now, this is quite a pain since now I have to install Mercurial from source so that it installs in my python\scripts folder. Why doesn’t the windows binary just install Mercurial directly to the Python folder on my machine? If py2exe is trying to do its own thing and using a separate Python interpret then maybe it shouldn’t be used because what if you want to add more modules that you need (which is exact my problem)? Then you have to bite the bullet and build from source. I’m sorry to say but that’s not very developer friendly. I mean there should be an easier way to add more modules to Mercurial

Building Mercurial from Source

I used the python setup.py method to build Mercurial. I followed the instructions here. So I downloaded the source and installed Visual C++ 9 (python extensions require a C compiler). Then I ran python setup.py build from the top level directory of the Mercurial Source. Text rushed through my console and BAAM! An error!

Turns out you need to use the same C compiler to build python extensions as the one you used to build python itself. Awesome, well my particular python build used Visual Studio 2003 (I just installed it from an exe, I didn’t build it myself). I can’t use that so I need to find another way. Well the documentation that I was using provided some help as to how I could use another C compiler.

After spending hours trying to get Mercurial to build with Visual C++, cygwin and then with mingw I somehow finally got it to work with mingw with the help of my friend, Tiago Moreira. This was a very tedious and unpleasant experience. My problems with Mingw were happening because I wasn’t installing it in my C: drive (since I have no available space) but it turns out that Mingw doesn’t like being installed in any drive that isn’t C: for whatever reason. Now after I miraclously cleared up space on my C: drive and snuck Mingw in there and then set the path in the windows environment variables my installation still didn’t work. Well, it turns out the path needs to be explicitly set on the command line for the build to work.

This experience hasn’t been very pleasant and I definitely think that python needs to figure out a better way to install extensions. I mean I should be able to use any C compiler I want with relative ease. Right now one has to go through a tedious process of configuring python to use the right compiler. If anyone out there can explain to me why that is, I’m all ears. The way that the documentation is written doesn’t help either. There is definate room for improvement when it comes to the clarity and detail of the document.

I know what some of you might be thinking at this moment. Why is this guy complaining about user friendliness and ease of use when talking about developer tools? They are not meant to be user friendly. Well, I say why not? I mean I am not asking for the program to hold my hand here but I would prefer if it didn’t work against me. I know some, maybe most of these troubles have croped up because of my inexperience with these technologies but that’s exactly my point. How can you get new people involved in your open source projects if there are frustrating road blocks in the way from the very beginning?

See, what I’m wondering is whats stopping Mozilla or the developers of Mercurial from making an installation bundle, which makes it easier to get a hold of all the dependencies. I know the Microsoft SDKs that Firefox needs to perform a build are proprietary. However, that could easily be handled if the installation bundle somehow pointed you to the download page and once the setup file is downloaded the bundle could then run the installation for you. I haven’t looked into the technical details of doing this but I would think that this would be possible. Maybe the developers have thought about this but haven’t had the time or the resources to implement it, I don’t know. I think creating an installation bundle would be a good idea, especially for beginners like myself. But nonetheless, enough of my ranting. I am not one to give up once I decide to take something on. I will continue working and putting in effort into these projects but I don’t know whether others will do the same.

All the troubles aside I finally got my installation working and ready to roll. I ran hg serve from the hgpoller repo and VOILA! No more errors, pushloghtml was FINALLY up and running. Now I can actually start working on my project

hg serve, no errors!

hg serve, no errors!

, ,

No Comments