Archive for October, 2008

FSOSS 2008 - Thoughts and Experiences

Posted by Sid in Open Source on October 24th, 2008

Today was the first day of the annual FSOSS (Free Software and Open Source Symposium) held at Seneca College, Toronto. The day began with registration which started at 8:15am. I got there at around 8:50 (just in time since the first round of presentations were slated to begin at 9am). Each person attending the conference was given a Seneca handbag with various things inside (including an FSOSS T-shirt, schedule booklet and the latest Ubuntu CD). The last couple of years there has been at least one key note speaker but that wasn’t the case this time around. Everybody was ushered into the various tracks and all the presentations began immediately.

Software Security

This time around the first day had four tracks instead of the usual three. This new track carried the same theme throughout as various Seneca people gave their thoughts relating to open source. I chose not to attend any of these tracks since I already have some idea of what they were going to talk about. The very first presentation that I attended, The Most Important Thing - How Mozilla Does Security, and What You can Steal was excellent. The speaker, Johnathan Nightingale was clear and confident, he knew his stuff. I found it interesting how he called browsers “stupid” since they download unknown code 3rd party and run it. I agree with him, it is “stupid” but that is what makes a browser so useful. The ability to run other people’s code situated anywhere on web makes the browser the most used application on a person’s desktop. Throughout the presentation Johnathan kept emphasizing the importance of writing tests. I think everybody agrees that they are needed, it’s just that it’s difficult finding people who want or like writing them. Tests are one of those things that are essential but are always hated. They are not fun or “cool” to most people. They are like taking medicine, we might hate the procedure but the results are beneficial in every way. The presenter stated that the most important thing is to learn from your mistakes so that you never make them again. I think this can’t be stated enough, even if I bolded, underlined and capitalized that statement it wouldn’t be enough. Such a statement doesn’t just apply to software security, it applies to life itself. I know that it has been repeated many many times over the years but people still don’t seem to get it. Albert Einstein once said, “The definition of insanity is doing the same thing over and over and expecting different results”. History repeats itself and learning from it is essential if you are to succeed.

Open Source Communities

I enjoyed the security presentation (see above) immensely, it was one of my favorite this year but I have to say that the last presentation that I attended, Community Building and the Architecture of Participation was absolutely brilliant. Greg DeKoenigsberg was captivating and expressive. He put forward an important question that I think many of us at FSOSS take for granted. He asked whether free software was better? This got me thinking. We all assume that it is better just because it’s free and open. The presenter said that he thought the answer to his question was yes, it is better. But I don’t know if he was clear enough with his question. What does he mean by “better”? Better in price? Better in quality? Better in service? In many of these categories free software can sometimes leave the user disappointed. Nonetheless, I think the presenter is right that free software is better, definitely from a philosophical and ethical standpoint at the very least. Now, the goal of every free software practitioner and supporter should be to make it better in every other way possible, but that is easier said then done. Another interesting point that came up during the talk was the way to get new contributors involved in your open source project. The presenter expressed that fact that it is important that one must allow people to start from the “edges” and work towards the “core”. Now what does this exactly mean? Well, many times open source software can be very challenging to get involved in. Many communities fail to provide a starting point that pushes new users in the right directions and allows them to learn more about the project. Basically, when a project fails to provide areas that beginners can easily work on the barrier of entry is increased exponentially and the likelihood of that project getting new users that eventually contribute significant knowledge decreases drastically. I wholeheartedly agree with Greg on this issue. It is a major problem with some of the open source projects out there today. How can this be fixed? How can projects make it easier for beginners to get involved? Couple of things that will definitely help are clear and concise documentation and helpful community members. It is important to let new users know that they are desired and that the work they are doing is important (even if it might not be). Attracting fresh recruits is a constant challenge for any organization or project and it is important to handle it correctly to secure the future of the project.

In the End…

Overall, FSOSS was an incredible learning experience for me, as always. The speakers mostly did a great job (excluding one talk, which I will not name) and I enjoyed myself throughtout the conference. I was glad to see more Seneca students come out to than conference than I have in the past two years so that is always good. I wish more students would take advantage of this great opportunity of having such a reputable conference right in our back yard. I wish I could have attended all the talks but alas that was impossible. I look forward to when the videos for each talk are released. The only thing missing from this years conference was a key note speaker. Nonetheless it was a great conference and I would like to thank all the volunteers and faculty that made it possible. I keenly look forward to FSOSS 2009.

, , ,

No Comments

Pushloghtml Should Show More Than 10 Entries, Working on v0.1 Release

Posted by Sid in Mercurial Project, Open Source on October 18th, 2008

Well as I stated in a previous blog post, one of my tasks for v0.1 release would be to make pushloghtml show more than 10 entries. Jorendorff, recommended that it show all the entries at once. Now, I took that to mean all the entries should be loaded onto the page on pageLoad because I assumed that the total push log entries couldn’t be more than a couple of hundred. Well, it turns out the number can be in the thousands and trying to load all of that on pageLoad completely kills the load time. I realized right there and then that this method would not work. Something else needed to be done.

I talked to jorendorff and ted about this issue and they said that the pushlog shouldn’t load all the changesets at once. They want more entries to be loaded on scroll down. Seems like there was some what of a miscommunication that took place, but no worries this sort of stuff can happen.

Now, I have never implemented this kind of feature so I had to figure out how to do this. It was recommended that I use jQuery, which allows one to measure the height of the page and the window. Thus, by the doing the following one can know when the user reaches the bottom of the page…

1
2
3
4
5

$(window).scroll(function() {
  if($(window).scrollTop() == $(document).height() - $(window).height()) {
    renderMorePushLogResults();
  }
});

The function renderMorePushLogResults() should display the next set of changesets on the page. Now, what does this function do to retrieve the correct data? Well, currently I am using a python script that returns data in JSON format. To access the data and show it on the page I can make an XMLHttpRequest to the JSON script.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

function renderMorePushLogResults() {
  var pushCheckins = new XMLHttpRequest();
  pushCheckins.open('GET', '/json-pushes?startID=0&endID=20', true);
  pushCheckins.onreadystatechange = function() {
  if(pushCheckins.readyState == 4)  {
    if(pushCheckins.status != 404) {
      var pushData = new Function("return " + pushCheckins.responseText) ();
      var divScroll = document.createElement("div");
      for(var i = 1; i <= 20; i++) {
        divScroll.innerHTML += pushData[i].user + "n";
        document.getElementById("titlePush").appendChild(divScroll);
      }
    }
   }
  }
  pushCheckins.send(null);
}

The code above gets the data from the script, creates a div, loops through all the user names provided by the the python script and puts in the div. Then the div is appended to the table named “titlePush”. You can see the result that the above 2 functions give http://sidkalra.com/files/mercurial/xmlhttprequest_example.png

Now, obviously this is not what I want. There are some issues that need to resolved…

  • The results are not formatted properly.
  • The python script returns two more types of data, the date and the changesets which I have chosen not to show on the page for now. However, it doesn’t return the patch author name and commit message.
  • The new changesets need to be printed on the page in the right sequence (by date) and the script doesn’t seem to return data sequentially.

As I make progress regarding the 3 issues I have pointed out above I will continually update this post so stay tuned!

Edit: Well I’ve been working on this the whole day. I tried using another script that ted recommended (http://hg.mozilla.org/mozilla-central/pushlog?fromchange=aa4b3566d739) since I was having issues with the first one (see above). The following is the code…

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

function getMoreChangesets() {
  try {
    var pushCheckins = new XMLHttpRequest();
  } catch(e) {
      alert("error with XMLHttpRequest object");
    }
  pushCheckins.onreadystatechange = function() {
    if(pushCheckins.readyState == 4)  {
      if(pushCheckins.status == 200) {
        var pushData = pushCheckins.responseXML;
 
        var element = "";
        for(var i = 1; i <= 10; i++) {
          element += pushData.getElementsByTagName('tr')[i].getElementsByTagName('i')[0].firstChild.nodeValue; //Read the first element
          element += ' | ' + pushData.getElementsByTagName('tr')[i].childNodes[1].firstChild.nodeValue;
          element += ' | ' + pushData.getElementsByTagName('tr')[i].getElementsByTagName('cite')[0].firstChild.nodeValue;
          element += ' | ' + pushData.getElementsByTagName('tr')[i].childNodes[2].firstChild.lastChild.nodeValue;
        }
 
        var divScroll = document.createElement("div");
        divScroll.innerHTML += element;
        document.getElementById("titlePush").appendChild(divScroll);
      } else {
          alert("Error with xml" + pushCheckins.status);
        }
    }
  };
  pushCheckins.open('GET', '/pushlog?fromchange=aa4b3566d739', true);
  pushCheckins.overrideMimeType('text/xml');
  pushCheckins.send(null);
}

The above code gives the following result on scroll down…

http://sidkalra.com/files/mercurial/xmlScript.png

Since this new script returns XML I am using pushCheckins.responseXML to retrieve the XML data. Then I can access all the data I want and push it on the screen (see line 14-18). Now, I still get the same issue of the formatting not working properly but at least this script provides with me with more data that I need, although it doesn’t give me everything like the email address of the author.

Edit 2: I’ve continued to work on this issue and I’ve come up with a new iteration of code that provides some better formatting for the output i.e. everything is not a blob of unreadable text like my above example. But the new changesets still don’t match the styles and formatting of the original ones. The following is the code…

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

function getMoreChangesets() {
  try {
    var pushCheckins = new XMLHttpRequest();
  } catch(e) {
      alert("error with XMLHttpRequest object");
    }
  pushCheckins.onreadystatechange = function() {
    if(pushCheckins.readyState == 4)  {
      if(pushCheckins.status == 200) {
        var pushData = pushCheckins.responseXML;
 
        var element = "";
        var divScroll = document.createElement("table");
        var test = pushData.getElementsByTagName('tr');
        for(var i = 1; i <= 10; i++) {  
          element += '<tr><td>' + pushData.getElementsByTagName('tr')[i].getElementsByTagName('i')[0].firstChild.nodeValue + '</td>'; //Read the first element
          element += '<td>' + pushData.getElementsByTagName('tr')[i].childNodes[1].firstChild.nodeValue + '</td>';
          element += '<td>' + pushData.getElementsByTagName('tr')[i].getElementsByTagName('cite')[0].firstChild.nodeValue + '</td>';
          element += '<td>' + pushData.getElementsByTagName('tr')[i].childNodes[2].firstChild.lastChild.nodeValue + '</td>';
          element += '</tr>';
 
          divScroll.innerHTML += element;
        }
        divScroll.innerHTML += "</table>";
        document.getElementById("newItems").appendChild(divScroll);
      } else {
          alert("Error with xml" + pushCheckins.status);
        }
    }
  };
  pushCheckins.open('GET', '/pushlog?fromchange=aa4b3566d739', true);
  pushCheckins.overrideMimeType('text/xml');
  pushCheckins.send(null);
}

You can see the result of the above code at http://sidkalra.com/files/mercurial/formattedXmlScript.png

As you can see from the example the formatting and readability has become much better but I am getting some repetition. I don’t quite know why that is happening. The problem is obviously between lines 16 and 20. Keep watching for more updates as I figure this out.

, , , ,

No Comments

Applying a patch to a Repo using Mercurial

Posted by Sid in Mercurial Project, Open Source on October 14th, 2008

As I mentioned in a previous blog post, one of the features for my v0.1 release will be implementing a expand/collapse feature for pushloghtml. Well, turns out that ted has already started working on that. He’s already filed a bug and released a patch for it. According to him its not quite done and a couple of things need to be changed.

So, my first job was to download the patch and apply it to my repo. So I headed over to Bugzilla and looked up the bug. There were 2 patches that I needed to download…

  • One for pushlog-feed.py
  • Another for gitweb_mozilla

Before you read the following take into account that I’ve never applied any kind of a patch before so this is purely from the point of a beginner. So I was looking for a way to download the patch so that I could apply it. But apparently there is no download link available on BugZIlla (it took me almost 20 mins to realize that). They should really look to put in some sort of download link/button. In the end I just ended up using the crude method of “Save Page As” to download the patch file.

Now onto applying the patch. I had no idea how to do that so I tried using tortoiseMerge since I already had that installed. It is apparently capable of applying a patch. I added the path of the diff file and the directory where I wanted it to apply the patch in the two text boxes below and then clicked the OK button. Nothing happened at all. Nothing got loaded, no error message, no nothing. This had me a bit bewildered and frustrated. tortoiseMerge was basically useless.

Now onto the next option which was hg itself. I knew that there was a way since I had read about it in some documentation when I was getting know hg at the start of my project. So I headed over to google to perform a search. I tried various things such as “apply a patch hg”, “patch hg”, “install patch hg” and “mercurial apply patch”. The last one gave me a couple of good results. One was at hgbook and another was documentation about hg queues but neither of them was a straight up guide of how to install patches. I tried many different google searches but I just couldn’t find any simple and easy guide explaining how one would install a patch using hg. Well, here I am to the rescue. The following is a simple guide that gives instructions on installing a simple patch using hg…

  1. Download the .patch or .diff file onto your machine
  2. Navigate to the repo that you want to install the patch to i.e. testRepo (I will using repo name for this example, substitute it with your repo name when you are trying to apply a patch yourself)
  3. Open testRepo\.hg\hgrc (the config file)
  4. Add the following to hgrc and save (This enables queues for hg so that we can use commands that we need, qimport and qpush)[extensions]
    hgext.mq =
  5. Now open up a command prompt and navigate to your repo directory
  6. Type in hg qimport <full-path-of-patch-file> and press enter
  7. The above command should create a patch folder within testRepo\.hg\patches\
  8. Next navigate to  testRepo\.hg\patches\
  9. Type in hg qpush <patch-file-name>
  10. Done! The patch should have been applied (if you get an error such as “abort: local changes found, refresh first” then that means you have made changes to the original files and the patch won’t work)

In my trials and tribulations while trying to apply a patch I had to first figure out what hg queues did and how to use them. I tried to just type in hg qimport but I just got a “command does not exist” error. I didn’t realize that I had to import something until ted gave me a push in the right direction. A simple guide would have really helped.

, , ,

3 Comments

Getting pushloghtml Up and Running Part Two

Posted by Sid in Mercurial Project, Open Source on October 12th, 2008

Carrying on from where I left off , my problems with python modules continue. After some conversations with djc, bsmedberg, ted and jorendorff it seems that Mercurial is using an entirely different python interpreter and that is why even though my 2 modules are installed properly hg serve is still throwing me an error.

I installed my Mercurial using a windows binary and it seems that it is based on py2exe, which zips up its own libraries and installs them separately along with a separate Python interpreter. Now, this is quite a pain since now I have to install Mercurial from source so that it installs in my python\scripts folder. Why doesn’t the windows binary just install Mercurial directly to the Python folder on my machine? If py2exe is trying to do its own thing and using a separate Python interpret then maybe it shouldn’t be used because what if you want to add more modules that you need (which is exact my problem)? Then you have to bite the bullet and build from source. I’m sorry to say but that’s not very developer friendly. I mean there should be an easier way to add more modules to Mercurial

Building Mercurial from Source

I used the python setup.py method to build Mercurial. I followed the instructions here. So I downloaded the source and installed Visual C++ 9 (python extensions require a C compiler). Then I ran python setup.py build from the top level directory of the Mercurial Source. Text rushed through my console and BAAM! An error!

Turns out you need to use the same C compiler to build python extensions as the one you used to build python itself. Awesome, well my particular python build used Visual Studio 2003 (I just installed it from an exe, I didn’t build it myself). I can’t use that so I need to find another way. Well the documentation that I was using provided some help as to how I could use another C compiler.

After spending hours trying to get Mercurial to build with Visual C++, cygwin and then with mingw I somehow finally got it to work with mingw with the help of my friend, Tiago Moreira. This was a very tedious and unpleasant experience. My problems with Mingw were happening because I wasn’t installing it in my C: drive (since I have no available space) but it turns out that Mingw doesn’t like being installed in any drive that isn’t C: for whatever reason. Now after I miraclously cleared up space on my C: drive and snuck Mingw in there and then set the path in the windows environment variables my installation still didn’t work. Well, it turns out the path needs to be explicitly set on the command line for the build to work.

This experience hasn’t been very pleasant and I definitely think that python needs to figure out a better way to install extensions. I mean I should be able to use any C compiler I want with relative ease. Right now one has to go through a tedious process of configuring python to use the right compiler. If anyone out there can explain to me why that is, I’m all ears. The way that the documentation is written doesn’t help either. There is definate room for improvement when it comes to the clarity and detail of the document.

I know what some of you might be thinking at this moment. Why is this guy complaining about user friendliness and ease of use when talking about developer tools? They are not meant to be user friendly. Well, I say why not? I mean I am not asking for the program to hold my hand here but I would prefer if it didn’t work against me. I know some, maybe most of these troubles have croped up because of my inexperience with these technologies but that’s exactly my point. How can you get new people involved in your open source projects if there are frustrating road blocks in the way from the very beginning?

See, what I’m wondering is whats stopping Mozilla or the developers of Mercurial from making an installation bundle, which makes it easier to get a hold of all the dependencies. I know the Microsoft SDKs that Firefox needs to perform a build are proprietary. However, that could easily be handled if the installation bundle somehow pointed you to the download page and once the setup file is downloaded the bundle could then run the installation for you. I haven’t looked into the technical details of doing this but I would think that this would be possible. Maybe the developers have thought about this but haven’t had the time or the resources to implement it, I don’t know. I think creating an installation bundle would be a good idea, especially for beginners like myself. But nonetheless, enough of my ranting. I am not one to give up once I decide to take something on. I will continue working and putting in effort into these projects but I don’t know whether others will do the same.

All the troubles aside I finally got my installation working and ready to roll. I ran hg serve from the hgpoller repo and VOILA! No more errors, pushloghtml was FINALLY up and running. Now I can actually start working on my project

hg serve, no errors!

hg serve, no errors!

, ,

No Comments

Bugzilla - Watch a User Feature

Posted by Sid in Open Source on October 7th, 2008

As a part of an exercise humph asked us to fiddle around with bugzilla. In this case, watch another user. Now what does that mean? That means that any bugmail that the user you are tracking receives will get forwarded to you. You can set this by

  1. going to bugzilla
  2. logging in
  3. going to preferences
  4. adding a person to your watch list.

This basically gives you an idea of what kind and amount of bugmail that that person receives. It puts you in their shoes.

I decided to track bsmedberg since he seems to be a busy man. So I “watched” him for about a week and got around 70 bugmail during that time. Thats quite a lot of traffic for just one week. It definitely gives me an idea of the amount of work bsmedberg has to deal with on a weekly basis, and this is just email.

This bugzilla feature gives me an idea of how someone like bsmedberg (someone that has a more important presence in the Mozilla community than myself) deals with the various kinds of bugs that he recieves on a regular basis. I’d recommend this feature to any novice or beginner looking to get involved with tracking or fixing bugs.

,

3 Comments

Getting pushloghtml Up and Running Part One

Posted by Sid in Mercurial Project, Open Source on October 5th, 2008

The past 2 days I have been struggling to get pushloghtml working. As I had mentioned in my previous post I needed 2 modules to do this, simplejson and pysqlite2. Turns out both of them are python libraries, which bsmedberg graciously pointed out to me.

Finding and Installing simplejson and pysqlite2

First thing I had to do was to install Python on my machine. I went with version 2.6, which was just released on Oct 1st (eventually I uninstalled it and used v2.5.2 since nothing was compatible with 2.6 yet). Then I hunted down and found simplejson’s and pysqylite2’s source. Now, I didn’t know what to do from here. How do I install them? I didn’t know. I laboured through various google searches but no avail. But then I remembered what bsmedberg had told me, use ez_install.py to install these libs. So I hunted down and found the file on this blog (there doesn’t seem to be an official place where this is housed). I downloaded and installed it on my machine.

One can give Easy_install python eggs to install packages/modules. Now I needed to find simplejson and pysqlite2 in egg format, which I found at the CheeseShop (or PyPi). I ran both the files with easy_install but all it did was put the respective egg files in  my \Python252\Lib\site-packages directory. Does that mean it installed properly? I don’t know. Ohhh, the challenges of being a novice.

I tried installing the exe for pysqlite2 to see if it does the same thing as easy_install (put an egg file in my python directory). It didn’t do that. It added

D:\CRULSHORUKH_Mobile\Apps\Python252\Lib\site-packages\pysqlite2

I guess that this means easy_install didn’t install simplejson correctly. I was sure now that pysqlite2 had been installed properly so I tried running hg serve to check whether the “no module found” error was gone for pysqlite2 at least. No luck, the error was still there.

What is the best way to know if a package installed correctly? Well, just use it! How simple eh? So I decided to create the following hello world program

import pysqlite2, simplejson
 
print "Hello World"

The result? No error! The program printed Hello World onto the screen, which means that the packages did install correctly! Now, this got me really confused. I have the packages that I need on my computer but why am I still getting the “no module found” errors?

More on this in Part Two…

, ,

No Comments

Deciding on v0.1 Release

Posted by Sid in Mercurial Project, Open Source on October 2nd, 2008

ETime is expeditiously approaching for me to decide on what I will be doing for my v0.1 release. The deadline is Oct 13th. As I mentioned on a previous post jorendorff had made a great blogpost on my project asking for people to brainstorm ideas to improve hgweb.They have been summarized on the project page. I think most of the ideas on there are not ideal for a v0.1 release. It would be wiser to do something simpler to start off and then go from there.

Before I think about starting on 0.1 I need to setup a development environment for myself. This involves getting gitweb_mozilla, buglink.py and pushloghtml all installed and working. I have the first two up and running now. As far as pushloghtml goes it is a database. To be frank, I don’t know much about it and I’ll have to query one of the big guys (jorendorff, ted, bsmedberg or djc) for some help. I have gotten a hold of it thanks to ted and aravind. Ted pointed me towards a README file in hgpoller that shows how to get the pushlog working. Basically I have to add a couple of lines to the the .hg\hgrc file (as I suspected). I tried that but I’m getting the following errors…

hgwebjson.py: No module named simplejson
pushlog-feed.py: No module named pysqlite2

In my email conversations with jorendorff I had asked him for some suggestions as to what I could do for 0.1 and the following is what he recommended…

  • Make pushloghtml show more than 10 entries (try to make it scroll forever)
  • Give large pushes in pushloghtml an expand/collapse button (Pushes can contain many changesets)
  • Make pushloghtml show which files each changeset touched

Now I am at liberty to do anything I want so I had a look at the ideas that jorendorff’s blog generated and the following caught my eye (keep in mind that I don’t know if any of these are too much or too less for a v0.1 release)…

  • Show more info about each changeset in an unobtrusive way (what information? I don’t know)
  • Be able to view more than 10 entries at a time in the changelog

I think out of all the the ideas I have listed I will most probably will be taking on the task of making pushloghtml show more than 10 entries. It seems quite straightforward although keep in mind this is coming from someone who isn’t quite familiar with the source code yet. Basically my tasks are

  • to get pushloghtml working on my machine
  • to locate the source files that dictate the amount of pushlog entries that get displayed and then alter it to display more entries.
  • to release v0.1 on time

EDIT: I asked humph to read this post and ok my v0.1 release agenda. He recommended that I take on both the tasks of allowing more than 10 entries for pushloghtml and adding an expand/collapse feature since they seem to go together. So there we go my list is final for v0.1 release…

  • to get pushloghtml working on my machine (some progress made on this part)
  • to locate the source files that dictate the amount of pushlog entries that get displayed and then alter it to display more entries
  • to add code that allows expand/collapse functionality (apparently all I need is a js lib according to humph)
  • to release v0.1 on time

Lots of work ahead!

, ,

No Comments