Posts Tagged Python
v0.6 Release - Removing the Page Navigation Links from Pushloghtml
Posted by Sid in DPS911, Mercurial Project, Open Source on February 21st, 2009
I’ve decided to remove the navigation links that appear at the top and the bottom of pushloghtml. This wasn’t part of my goals for this release but I’ve decided to add this at the last minute. The patch is for bug 459727, which loads more data OnScroll and thus makes the navigation links obsolete. There is no need for them to be there anymore.
Removing them was pretty simple. All I had to do was remove the following code from hg_templates/gitweb_mozilla/pushlog.tmpl
Page #changenav%navpushentry#I had to remove the above line of code from two places to get rid of both the top and bottom navigation links. Taking these navigation links out removes the unneeded clutter from the page.
v0.6 Release - Refactoring to Fix the Bitrotting Issue with Bug 459727
Posted by Sid in DPS911, Mercurial Project, Open Source on February 21st, 2009
I had mentioned in my previous blog post that hgpoller/pushlog-feed.py had bitrotted. One of my goals for this release was to make changes to the current version of pushlog-feed.py so that my patch is no longer broken for bug 459727. I’ve finally made those changes, which mainly occur in pushes_worker(). The following is what this method looks like with my changes:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 | def pushes_worker(query, repo): """Given a PushlogQuery, return a data structure mapping push IDs to a map of data about the push.""" pushes = {} for id, user, date, node in query.entries: mergeData = [] ctx = repo.changectx(node) if len(ctx.parents()) > 1: for cs in ctx.parents(): mergeData.append(hex(cs.node()) + '|-|' + clean(person(cs.user())) + '|-|' + clean(cs.description())) if id in pushes: # we get the pushes in reverse order pushes[id]['changesets'].insert(0, node) pushes[id]['mergeData'].append(mergeData) else: pushes[id] = {'user': user, 'date': date, 'changesets': [node], 'formattedDate': util.datestr(localdate(date)), 'individualChangeset': hex(ctx.node()), 'author': clean(person(ctx.user())), 'desc': clean(ctx.description()), 'mergeData': mergeData, 'max': gettotalpushlogentries(conn) } return pushes |
Basically I had to pass in repo (web.repo) so that I could have access to repo.changectx(node). This now allows me access to ctx.parents() which I need to retrieve merge changeset data. I also went through the whole file and changed every instance where pushes_worker() was called so that repo was being passed in as a paramater along with query.
These are all the changes I needed to make to the server side code. Now I’ll have to examine the changes that were made to the client side which caused my patch to bitrot.
v0.6 Release - Examining the Changes with hgpoller/pushlog-feed.py
Posted by Sid in DPS911, Mercurial Project, Open Source on February 20th, 2009
I had mentioned in my v0.6 goals blog post that my patch for bug 459727 had bitrotted. Unfortunately significant changes were made to hgpoller which apparently broke my patch. I need to remedy this situation because all my server side functionality for this patch is in the pushlog-feed.py file.
I downloaded the latest hgpoller source code and had a look at the changes that had been made. The file has changed in quite a few places. It seems that the function that my patch alters has been changed as well. I’m talking about pushes_worker(), which is repsonsible for passing the data from the server side to the client side.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | def pushes_worker(query): """Given a PushlogQuery, return a data structure mapping push IDs to a map of data about the push.""" pushes = {} for id, user, date, node in query.entries: if id in pushes: # we get the pushes in reverse order pushes[id]['changesets'].insert(0, node) else: pushes[id] = {'user': user, 'date': date, 'changesets': [node] } return pushes |
Now, the problem is that I need access to web.repo within pushes_worker() so that I can call repo.changectx(node) but right now, I don’t have access to repo within the method. I’ll have to figure out a way to do that somehow.
v0.6 Release Goals
Posted by Sid in DPS911, Mercurial Project, Open Source on February 15th, 2009
It’s time to start working on my 3rd release for this semester. My goal is to put out 3 patches this time around:
- A patch for the Mercurial Project’s Paper theme (fix for hg annotate, similar to what I did in v0.5)
- A small fix to bug 445560 so that it can finally go live
- A patch to update pushlogfeed.py for bug 459727 since it seems that the code has bitrotted and is no longer acceptable.
These 3 patches should combine to make a good solid release. These are 3 seperate improvements to hgweb that I’m sure users must be looking forward to having. I’m not quite sure what changes have been made to pushlogfeed.py causing it to bitrot. The solution to the problem may be simple or it may be complicated. I was under the impression that no changes needed to be made but new code has been added causing my code to break. Another exciting prospect is that I’ll have my name added to the Mercurial Project once I implement the annotate fix for the paper theme making my work available to all hgweb users.
I’ll be putting all the details of my work on the project page and this blog. Time to get to work!
v0.3 Release - Fixing the Problems with my Patch for bug 459727
Posted by Sid in Mercurial Project, Open Source on December 3rd, 2008
I’ve been busy trying to fix problems that had arised regarding the patch I submitted for bug 459727. All that is being done right now is fixing various small problems but the overall functionality remains the same. So the following are the solutions to the problems that jorendorff identified:
To calculate the max SELECT COUNT(*) FROM … should be used
Changed my implementation to use gettotalpushlogentries(conn) which performs a select statement. I’ve gotten rid of getMaxEntries() since it is no longer required.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 | def pushes_worker(repo, startID=0, endID=None): stmt = 'SELECT id, user, date, rev, node from pushlog INNER JOIN changesets ON id = pushid WHERE id > ? %s ORDER BY id ASC, rev ASC' args = (startID,) if endID is not None: stmt = stmt % 'and id <= ?' args = (startID, endID) else: stmt = stmt % "" if os.path.basename(repo.path) != '.hg': repo.path = os.path.join(repo.path, '.hg') conn = sqlite.connect(os.path.join(repo.path, 'pushlog2.db')) pushes = {} for id, user, date, rev, node in conn.execute(stmt, args): mergeData = [] ctx = repo.changectx(node) if len(ctx.parents()) > 1: for cs in ctx.parents(): mergeData.append(hex(cs.node()) + '|-|' + clean(person(cs.user())) + '|-|' + clean(cs.description())) if id in pushes: pushes[id]['changesets'].append(node) pushes[id]['mergeData'].append(mergeData) else: pushes[id] = {'user': user, 'date': date, 'changesets': [node], 'formattedDate': util.datestr(localdate(date)), 'individualChangeset': hex(ctx.node()), 'author': clean(person(ctx.user())), 'desc': clean(ctx.description()), 'mergeData': mergeData, 'max': gettotalpushlogentries(conn) } return pushes |
`new Function(”return ” + entries.responseText) ()` is saying
`eval(entries.responseText)`, perhaps use `JSON.parse (entries.responseText)`
I haven’t implemented this yet because I want more information on why the way I have done is not the appropriate way. I’ve asked on the bug page (comment 12)but haven’t heard back yet
The page loads more results when the user scrolls. What if the initial results don’t fill up the window?
I added the following CSS to always show a scrollbar no matter what. This should allow the user to scroll even if the initial entries don’t fill up the page.
html { overflow: scroll; overflow-x: auto; }
getMaxEntriesis called, then ’start’ is used once the user scrolls. But the user might scroll before the first result comes back, while start is still zero — a race condition
I changed the implementation to check the value of the start variable before more entries are allowed to load onScroll. If the value of start is still 0 then no entries are allowed to load. The following is the code:
$(window).scroll(function() { if(start > 0) { if($(window).scrollTop() == $(document).height() - $(window).height()) { renderMorePushLogResults(); } } });
Better to move CSS and formatting to stylesheets where possible
I removed explicitly setting the styles of the rows. Now I am just setting the class name of the row so that it uses the preexisting styles for that class name. View lines 37-43 for the code
The JS code isn’t consistently indented
It should be correctly formatted now. View here
This patch seems to contain some non-ASCII characters. Bugzilla doesn’t render it properly
Used &mdash in the places where non-ASCII characters were being used. View lines 73 and 121
In a few places the b tag is used where the server uses the strong tag
Changed the b tags to use strong. View lines 73 and 121
I will be putting up a new patch very soon…
EDIT:
The new patches have been posted here
v0.3 Release - A Problem with Getting data for the Graph View
Posted by Sid in Mercurial Project, Open Source on December 1st, 2008
hgweb has a graph function found in mercurial/hgweb/webcommands.py which returns a template. It also returns a variable called jsdata which contains all information that I want to display on the page along with the graph. Now, my problem was that I wasn’t quite understanding how I can could call graph() so that I could pass the data it returns to the client side.Should I have a new template or can I pass the data to an existing template.
For the longest time I was stumped on this issue. I didn’t understand whether I should be calling this function in my python code or could I call it on the client side using JavaScript? Did I have to play around with the template’s map file? I guess my lack of experience working with web applications that use templates and themes was rearing its ugly head. I was stumped!
I tried various methods via trial and error to see if I could figure it out but all in vain. Eventually I decided it was time for some help. I asked djc who told me that I would have to create a new template file which gets its data from the graph function in webcommands.py . But I still didn’t get how to actually pass the data to the client side.
So, I decided to ask for help in freenode’s mercurial channel. There, Brendan was able to point out that all I needed to do was add the following to hg_templates/gitweb_mozilla/map…
graph = graph.tmpl
The answer was staring me right in the face all along. I had glanced over this line many times while trying to figure out how this works but I guess I just didn’t realize that this is how you create a new tmpl file. Now I can access jsdata by adding {jsdata|json} in my template file.
Now I can actually start working on rendering the graph on the client side.
v0.3 Release - Understanding djc’s Graph View
Posted by Sid in Mercurial Project, Open Source on November 25th, 2008
I’m starting work on a graph view for the pushlog. Frankly, I didn’t know where to begin since I have never really done anything like this before, then again I could say the same thing about the whole project. After having a discussion with jorendorff I was pointed towards djc’s (Dirkjan Ochtman) Graph view for Mercurial. He already has it working and my job is to now understand what he has done and get it working for the pushlog now.
The Source
I downloaded the source from http://hg.intevation.org/mercurial/crew/ using hg clone.
Getting it Running
Getting it running on my browser was pretty simple. Almost identical to what I had to do for the pushlog. Just open up a command prompt and navigate to the directory where you saved the source and call the hg serve command. Then fire up a browser and type in http://localhost:8000 and you should be able to see the graph view.
Understanding the Code
Mercurial crew comes with Mercurial and hgweb. Some of the functionality is implemented in crew/templates/static/graph.js which is what runs on the client side. Also the tmpl file used to render the page is crew/templates/coal/graph.tmpl which imports graph.js to utilize its methods. The template also takes in a parameter, jsdata which contains the graph data. jsdata is populated on the server side by the graph method of the crew/mercurial/hgweb/webcommands.py file. It contains graphmod.graph() which is apparently what does the actual graph layout.
I don’t fully understand this code at the moment. There are many questions going through my mind. For example it seems that the server side code (mercurial.hgweb.webcommands.graph) returns a tmpl so I don’t know whether I should be working from a new tmpl file or can I put my client side code in hg_templates/gitweb_mozilla/pushlog.tmpl like I have been doing all along?
The client side code receives the data from the server side through {jsdata|json} which is all the changeset data. But right now with the pushlog all this is processed through pushlog-feed.py which gives the data to hg_templates\gitweb_mozilla\map that renders the page. So should I be messing with this map file? If not then how can I pass the data to the client side so that I can draw the graph and render it on the page?
The above 2 examples are only some of the questions going through my mind right now. I have told you about them because they are ones that must be answered urgently. It is important that I understand the basic functionality of djc’s Graph view so that I can deduce a way to apply it to my problem.
It is still early days for this feature and I’m still trying to feel my way through it. Hopefully things will pick up as my understanding of this feature increases.
v0.3 Release - Fix for my Patch for bug 445560
Posted by Sid in Mercurial Project, Open Source on November 24th, 2008
In a previous blog post I detailed problems with my previous patch for bug 445560. There were 2 problems that ted identified which need to be fixed:
- Should use ids instead of dates
- Should store unique identifiers in the class instead of the id of the tag
The Fix
I made a change to pushlog-feed.py to use ids of the changeset instead of the dates by adding the following code which passes the id to the client-side (line 270):
248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 | def changelist(limit=0, **map): allentries = [] lastid = None ch = None l = [] mergehidden = "" p = 0 currentpush = None for id, user, date, node in entries: ctx = web.repo.changectx(node) n = ctx.node() entry = {"author": ctx.user(), "desc": ctx.description(), "files": web.listfilediffs(tmpl, ctx.files(), n), "rev": ctx.rev(), "node": hex(n), "tags": nodetagsdict(web.repo, n), "branches": nodebranchdict(web.repo, ctx), "inbranch": nodeinbranch(web.repo, ctx), "hidden": "", "push": [], "mergerollup": [], "Id": id } |
Then a change needed to be made to the hg_templates\gitweb_mozilla\map file. The id needs to be stored in the class of the hidden rows and the expand row. So I made the following changes:
mergehidden = '<br/>← #count# hidden changesets <a class="expand hideidentifier#Id#" href="#">[Expand]</a>'
pushlogentry = '<tr class="parity#parity# #hidden# identifier#Id#">...
Now each expand row will have the class hideidentifier#id#, which is a unique identifier telling us which expand link is clicked. Then we can take this unique identifier and hide all the rows that have the same #Id#. To do this I needed to add some JavaScript code to hg_templates\gitweb_mozilla\pushlog.tmpl…
30 31 32 33 | var id = $(this).attr("class"); id = '.' + id.substring(11, id.length); $(id).nextAll(id).toggle(); return false; |
So line 30 gets the id of the current object (the expand row whose [Expand] link is clicked by the user). So the id might be something like “hideidentifier2434″ after line 30 is executed. Next a ‘.’ character is added on the front and the id is substringed to give us something like “.identifier2434″. Now we have the class of the rows we want to hide/unhide and we can then execute line 32. Voila! Problem fixed.
v0.2 Release - Getting the Correct Chronological Order for bug 459727
Posted by Sid in Mercurial Project, Open Source on November 15th, 2008
Throughout my entire time working on a v0.2 release I haven’t been able to get the correct choronological order (it should be reverse chronological order) for the new entries that load OnScroll. This is an essential part of the pushlog as it shows users when the entries were added. I’m using a script called json-pushes to acquire the data I need to show. For each call to the script I need to provide a startID and endID. At first I started by calling the script like so:
- startID=0, endID=20
- startID=20, endID=40
- startID=40, endID=60
- startID=60, endID=80
- etc…
This didn’t work since since ID=1 is the very earliest entry in the pushlog when I should be starting with the later entries and working my way down to the earlier entries. So, what I needed to do was somehow acquire the ID of the latest entry in the pushlog. Thus, I wrote this new function in the server side file pushlog-feed.py
1 2 3 4 5 6 7 8 | def getMaxEntries(repo): stmt = 'SELECT id from pushlog' max = 0 conn = sqlite.connect(os.path.join(repo.path, 'pushlog2.db')) for id in conn.execute(stmt): max += 1 return max |
The above code provides me with the maximum ID in the pushlog. Now I need to provide this value to the client side so that I know which entry I should start off with when I make my first call to json-pushes. The following is the code:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | function getMaxEntries() { var entries = new XMLHttpRequest(); var max = 0; entries.open('GET', '/json-pushes?startID=0&endID=5', true); entries.onreadystatechange = function() { if(entries.readyState == 4) { if(entries.status != 404) { var entryData = new Function("return " + entries.responseText) (); max = entryData[1].max; start = max - 20; } } else return 0; } entries.send(null); } |
So it turns out that I needed to make two XMLHttpRequest’s to first acquire the maximum ID (see above) and then the next one to actually acquire all the data to show OnScroll. Thus, now everytime the user reaches the end of the page json-pushes loads the IDs with the following pattern:
- startID=[MAX], endID=[MAX-20]
- startID=[MAX-20], endID=[MAX-40]
- startID=[MAX-40], endID=[MAX-60]
- startID=[MAX-60], endID=[MAX-80]
- etc…