Friday, September 12, 2014

My history with Sakai

Tomorrow,  September 13 is the 10th anniversary of Sakai at UdL. We ran into production with Sakai 1.0 rc2 in University of Lleida in 2004. Quite an achievement and an adventure that has lasted 10 years and hopefully will be able to last many more. Perhaps it was a little rushed but luckily it worked out fine.

I will not tell you the history of the UdL and Sakai, I'll tell you what I know and I feel about my history of Sakai, that is directly related with the UdL. To get the full version of the UdL we should have a lot of people points of view. 

So I will start before Sakai, we have to go back few months ago, In January 2004 I applied for a contest for a temporary position at UdL for a project to provide to the University an open source LMS system. The tests were based on knowledge of programming in Java servlets, jsp,  and knowledge of eLearning. The IT service management was looking for a Java developer profile as they were evaluating the coursework platform from Stanford. They wanted developers to make improvements and adapt it to the UdL needs. At that time, UdL ran WebCT and wanted to replace it to an open source one in the context of free software migration across all the University.


I had coded a little Java for my final degree project, but I didn’t know anything about servlets or jsp, so I bought a book of J2SE and I studied some days before and took the test with many other they wanted that position. I passed the tests, and I was lucky to win the programmer position  on the “Virtual Campus” team  with 2 other guys.  David Barroso was already the analyst programmer of the team, it mean he was my direct boss (a really good one).  

We ran a pilot with a few subjects with the Computer Science degree in Coursework, and it looked to be well adapted to our needs. Also we were looking closely the LMS CHEF. When founding universities of Sakai announced that they join to create an LMS based on the work of those LMS the decision was taken.

When Sakai started lacked many features that we thought necessary, like a gradebook, a robust tool for assignment  and assessment,  but still seemed a platform with great potential. It had a big funding and support from the best universities in the world, it was enough for us to get into the project. UdL intention  with Sakai was to go beyond the capabilities of LMS and use it as a virtual space for the whole university. Provide in a future a set of community sites, and use for our intranet as well as development framework for our applications.

So we started working on it, translating the interface of Sakai to catalan and adapting the institutional image. We created sites for the subjects of the studies of the center Escola Politècnica Superior of the UdL. The September 13, 2004 the platform was in production.

Sakai 1.0rc2 translated in catalan and customized for UdL





During the process, we realized that the need of translating all the platform in each version would be very expensive, and the internationalization process appeared not to be one of the imminent community efforts, so the IT service manager Carles Mateu and David Barroso decided to offer support to internationalize Sakai. The idea was to provide a mechanism to translate Sakai easily without having to modify the source code every time Sakai released new version. It was an essential feature for us and it must be done in order to continue with Sakai project.  David contacted with the Sakai project chief director Dr. +Charles Severance and offered our help to internationalize whole Sakai.

Chuck was glad about our offering and the work started soon. Beth Kirschner was the person in charge of managing our work and sync with Sakai code. I was lucky to have the responsibility to manage the task on our part. First thing I did was a PoC with a tool. I extracted all the properties of a VM tool to a properties file, and then it was loaded with Java Properties objects. The PoC worked well but Beth encouraged me to use ResourceBundles instead of simple Properties class. I wrote another PoC with this guy and it worked great. From that point then began the tedious task of going all the code to do this. The result was  “tlang” and “rb” objects everywhere. That took between 2-3 months 3 people. We also used that process to write the catalan translation. We used a Forge instance installed at UdL to synchronize these efforts. We implement the changes there for Sakai 1.5 and when a tool was completely internationalized I notified in order for Beth to apply  the changes in the project main Sakai branch.

Although we worked on a 1.5, i18n changes were released in version the Sakai 2.0. For us it was a success because it ensured that we could continue using this platform for longer.  When version 2.0 came out we upgraded from our 1.0rc2. Only one word comes to my mind when I remember that upgrade: PAIN. We had a very little documentation and we had to look for the code for every error we found. We had to make a preliminary migration to 1.5, running scripts and processes on the Sakai startup and then upgrade to 2.0. The migration process failed on all sides but with a lot of efforts finally we went ahead with it.

Once we had the platform upgraded, we started to organize our virtual campus and university-wide LMS as intranet, creating sites for specific areas and services and facilitating access to people depending on the profile had in the LDAP. We also created the sites for the rest of degrees of our University. 

From that moment our relationship with Sakai has not been so hard. Everything went
better. Next version we ran was 2.2, we upgrade it on 2006. By then we  were granted with a Mellon Foundation award for our internationalization effort in Sakai. It’s one of the things that I’m prouder of my career, but it was embittered because the prize reward was finally not claimed. I did not find out until a couple of years after that happened.  The money of the award should be spent developing something interesting related on education, so in order to receive the award was needed to make a project proposal detailing how we would spend the $50K. UdL’s idea was to create a system to translate Sakai’s string bundles easily like some tools did it by then (poedit, ...). The IT service direction thought it was a better that the project wasn’t done by the same team that customized Sakai at UdL and internationalized (I guess they had other priorities in mind for us), but that development should be done by people of a Computer Science research group of the UdL. I do not know why they didn’t do the project or the proposal to get the award money, but nowadays I already don’t mind. [Some light here ... see the comments]

Around that time our team started working on a new project where were implied a large number of catalan Universities,  The campus project. It initially began as a proposal to create an open source LMS from scratch to be used by them. The project was lead by Open University of Catalunya, UOC. The UdL IT service direction board and David Barroso expressed their disagreement to spend the 2M€ finance such a project having already open source LMSs like Moodle and Sakai in which they could invest that money. The project changed direction and they tried to do something involved with existing LMSs, so they decided to create a set of tools that would use an OKI OSID middleware implemented for Moodle and Sakai.  Although running external tools in the context of an LMS using standards and provide an WS BUS to interact with the LMSs API was a good idea  I didn’t like how wanted to use a double level OKI OSID layer to interact with both LMS APIs. I thought that was too much complex and hard to be maintained.


OKI BUS

We upgraded Sakai again in 2007 to version 2.4 (that release gave us a lot of headaches). I also won the position as analyst programmer in the Virtual Campus team that David Barroso vacated when he won the internal projects manager position. The selection process left me quite exhausted by the long rounds of tests that were delayed in time and the competition with nearly 30 colleagues the made ​​harder the effort to get the best grades to win the position. By then, the IT service direction board, Carles Mateu and Cesar Fernandez, resigned because they had discrepancies with the main university direction board about how to apply the free software migration in the UdL. It was a shame because from then we have experienced a strong running back in free software policies and has worsened the situation of the entire IT service.

In September of that year, after the job competition finished and being chosen to take it, I went to spend a couple of weeks at the University of Michigan. My mission there was to work on the IMS-TI protocol with Dr. Chuck to see if we could use this standard as part of the Campus project. These two weeks there were very helpful. We did several examples implementing IMS-TI with OSID.  I spent a good time with Chuck and Beth in Ann Arbor during my visit to the United States, but I really remember fondly that trip because a few days before I went to Michigan, I got married in Las Vegas and I spent our honeymoon in New York.

Once back to Lleida, I insisted several times to the Campus Project architects on changing standards for the registration and launch apps to IMS-TI. Although people lead the campus project loved that idea they had already deep in mind the architecture they wanted to use, so we went with the original idea.

Several of the partner universities in the project created tools for that system and the UdL picked up the responsibility to create OSID implementations for Sakai as well as a tools to register and launch remote tools within Sakai as if they were their own . Although it was very tedious to implement OSID, it allowed me to get a fairly deep knowledge of all systems that later became the Kernel of Sakai. Unfortunately, the campus project was not used, but parallel IMS-LTI could end up winning.

Already on April 2008, taking advantage of a visit by Dr. Chuck to Barcelona for an attendance at a conference organized by the University Ramon Llull, we had the first meeting of Spanish Universities that had or thought to run Sakai. 

I went with the new director of IT services of the UdL, Carles Fornós. He was there the first time I saw Sakaigress, furry pink Sakai’s mascot. Dr. Chuck was carrying her. I explained to my boss that these teddies were given as a reward for participation in the community, and the first thing he told me was, "we have to get one." During the meeting the representatives of both universities that had running sakai, UPV as we, explained a bit of the experience we had with Sakai and resolved doubts that were raised to us from other universities. At the end of the meeting, everyone's surprise, Dr. Chuck wanted to give to us (UdL) the Sakaigress. He did it for two reasons that told me later. First, because we had been working hard in the community to internationalization and helping to promote standards like the IMS-TI with our work in  the implementation of the campus project, on the other hand he gave it to silence some voices of doubt that came out in the environment of our university about choosing Sakai instead of Moodle, wanting to reaffirm the commitment of the community with our University. 

Sakaigress

During that meeting also came the idea of making the first workshop of Sakai. A way to show people how to install, make tools and discuss about the platform. When my boss heard it whispered to me that we should offer volunteers to organize it, so I offered to organize.

In that meeting I also met the man who was in charge of implementing Sakai in the Valencian International University (VIU). We talked with him ​​about the OKI OSID implementation with his technical staff by mail some days before. They were very interested  in this use case. It was not even a month that  the team that prepared the specifications for implementation of Sakai to VIU came to Lleida to visit us. Before I tried to convince Carles Fornós to offer our services to the VIU. The customization of Sakai on other university for us would have been very simple and it was an opportunity to provide to the UdL more funds to keep developers. Carles did not seem a good idea, so I did not even offered. 
Moreover, when the UdL rejected to offer services as an institution, I considered doing at personal level with the help some co-workers. At first it seemed like a good idea for the responsibles for the technical office of the VIU, but when the moment arrived to go ahead with the collaboration, the UdL main direction board showed their disapproval (no prohibition), which made ​​us pull back because the risk of losing our jobs in the UdL if anything went wrong. Finally, the work was made by Pentec-Setival (Samoo). They did a great job. Perhaps it was the best result for the Spanish Sakai community because we got  a commercial  provider to support  Sakai.

In June 2008 we held the first Sakai workshop. It was a very pleasant experience, where the colleagues from UPV Raul Mengod and David Roldan, along with some staff of the Institute of Education Science of UdL (ICE) helped me to give some talks to other universities that were evaluating Sakai as their LMS.  

Soon after, in February of 2009, it was organized the second Sakai event in Santiago de Compostela. There, the group of the S2U was consolidated. By then, the UPNA was about to run on production migrating the contents of its old LMS WEBCT. In that meeting I showed how to develop tools in Sakai. At UdL we had upgraded to 2.5 and also shared opinions. We suffered a lot for performance issues and crashes with 2.4, but 2.5 seemed to improve a lot.

Days later that event, UPV invited us to attend a presentation and a meeting with Michael Korcuska, Sakai Foundation executive director by then. In Valencia it was the first time I saw the preview of Sakai 3. It was sold as the new version that would replace Sakai 2, he told that perhaps community would release a 2.7 version but not a 2.8. It was expected to be on 2010.

Truth be told, I loved it, and I spent much time tinkering and learning new technologies that had behind sakai 3. I went to the workshops offered at the 2009 conference in Boston, the truth is that everything pointed to the community supported the plan to move to Sakai 3, or at least it seemed to me.

On the 3rd congress of the S2U on November 2009, I made ​​a presentation of the benefits and technology behind Sakai 3 for making people aware of the new road that faced the LMS. Unfortunately we all know what has been the real way. It passed as slowly from “being the replacement”  to “something complementary” and finally to “something totally different”.

We did some proof of concept with hybrid system between Sakai CLE i OAE, Bedework, BBB and Kaltura. The PoC was quite promising, but the shift in architecture given the poor results obtained with the technological stack chosen frustrated our plans. Currently OAE continues with another stack but this away to the idea we had in mind at first.

By then we owned a big number of tools developed for Sakai JSF and Spring-Hibernate. For us, it was a problem in the future expected platform migration process between 2 and 3. In late 2009 and early 2010 we started developing our own JS + REST framework based on Sakai to have tools implemented more neutral manner that would allow us to move between platforms in less traumatic process. Thanks to all what I learned from Sakai OAE technologies I designed what is now our tool development framework for Sakai, DataCollector. It’s a framework that allows us to link to multiple sources and types of data sources and display it as js apps inside Sakai. It uses Sakai realms as permission mechanism and lets create big apps based on templates.
Gradually we have been replacing all the tools created in JSF (poor maintainable) by these based in our framework.  Although we finally we have not moved to OAE platform, it has helped us to have a set of more flexible and maintainable apps than those written  in JSF.

In July of 2010 we upgraded to version 2.7. We were still hoping to see soon Sakai OAE as part of our ecosystem Virtual Campus. Everything seemed to fit pretty well. At the 2010. At the end of the month my first son was born, and I took a long paternity. I was not working in the UdL but I wanted to assist to the IV Spanish Sakai congress in Barcelona in November to show all the work made with the Datacollector.  I went with my wife and him, the youngest member in the S2U.

In June 2011 we had another meeting in Madrid, it was organized to show to whole S2U member how to coordinate and use JIRA in a better way to help to our contributions being incorporated in sakai trunk. Some time ago we had an arrangement to implement some functionalities together and it was difficult to get in the main code. Some universities paid to Samoo to get it implemented but UM and UdL preferred to implemented ourselves. But what I really enjoyed in that meeting is how UM had implemented Hudson in their CI process. I loved the idea and my task in the following months was refactor all our process and automatize builds, deployments and tests with jenkins and selenium. 

Looking backward I see that during the years 2010 to 2012 our involvement with the S2U and the whole Sakai community dropped considerably. I guess that our eyes were on shift to the new environment. We concentrate our efforts on having DataCollector framework developed as much as possible in order to have a valid output gate for all those tools that have been developed since 2004. In addition S2U objectives were not in line with what we had in that moment. S2U's approach focused on the internationalization. As I understand it was a mistake because there was already a part of the community focused on that and the S2U should not focus only on those issues.

In July 2013 we did the sixth and last upgrade. In the upgrade process to  2.9 we took the chance to spend some time to migrate from our users provisioning system based scripts to an implementation of Course Management. Mireia Calzada did an excellent job  preparing ETLs and helping to build an implementation based on hibernate.

We took that opportunity to open all the functionality to allow create sites by teachers and students to let them use to work together, now they have storage space for their own projects, communication tools, etc.  That gave us very good results because people feel the virtual campus more useful than previous years. Also, we allowed teachers to invite external people and organize their sites as they want. Many of the complaints we had about the Sakai platform weren’t about features not supported by Sakai but due to the restrictions imposed by us.

The previous tasks related with that upgrade allowed me to reconnect with the community collaborating  reporting and resolving bugs, participating in QA,  and contributing what my colleagues and I have translated into catalan.

During 2013 I also ventured on a personal project related with Sakai. I created together with Juanjo Meroño from Murcia a functionality to allow videocam streaming  using the Sakai’s portal chat. A desire to contribute something personal to free software and especially to Sakai motivated me to make this project. It was a pretty nice experience to work with the community again. The help of Neal Caidin and Adrian Fish was the key to integrate it to the main code Sakai.  

In November 2013, Juanjo and me presented that functionality in the VI Congress of Sakai in Madrid. The important thing about that congress was that the whole s2u recovered sinergia. I’m convinced that University of Murcia staff was the key to inspire the rest of us. If you are interested you can read my opinion of the event in a previous blog post. Now we have weekly meetings and work as a team. Resources flow gently between the needs of group members and goes pretty well.

Now I feel again that I’m part of the Sakai community and S2U. I guess that the fact of working closely with its members has allowed me to believe that Sakai has a bit me.I'm waiting when the next s2u meeting is going to celebrate, and maybe I'm gonna go with my second son born that August.

And that is a brief summary of how I remember that history, maybe something was different, or happened in a different time. Just say Thanks to UdL, Sakai project, and S2U members to make that experience so amazing. 







6 comments:

  1. Just a little note. The proposal to claim the award was made, and sent. In fact, as far as I know, the Mellon Foundation sent the "check". Somehow (I still don't know how after all those years), the University economic management managed (pun!) to lose the check and never cashed it.

    The decision to move the price project execution to a research group was a "technical" one, as the best legal way to hire people (that was what the prize money was going to be used to) was via the R+D mechanisms to hire people (not as regular UdL staff).

    ReplyDelete
  2. Wow Carles, that's worse than I believed. Did they lose the check? WTF.

    Thanks for being part of this history.

    ReplyDelete
  3. This comment has been removed by a blog administrator.

    ReplyDelete
    Replies
    1. Thanks. It's time to reset the counter of the top contributions, we already got too much credit for that. Let's work hard again to achieve the future goals and keep Sakai awesome for the next 10 years.

      Delete
  4. Previous Dr. Chuck comment was deleted. Sorry. He said:

    Alex - What a wonderful post. It shows how open source is not just about getting a "free" product - it is about a community respecting one another and working hard for each other. Your contribution to Sakai 2.0 in early 2005 to internationalize Sakai before its first broad release was one of the top five contributions to Sakai over the past decade - so I thank you.

    ReplyDelete
  5. Alex, are you currently still involved in software localization projects? If yes, you should have a look at the localization tool https://poeditor.com/

    This online l10n platform has all sorts of automation features and allows collaborative online strings translation, so it may help you improve the overall translation and localization workflow.

    ReplyDelete