Pakistan General Elections were scheduled to happen on July 25th, 2018. Urdu team wanted something similar to Karnataka state election. The idea was to create a something which will have constituencies breakdown and live results along with additional graphics related to the big election.
We made a timeline keeping in mind that there are other projects to get out during this time too and we followed most of it :P
Before the election
Election Commission of Pakistan (ECP) announced that they have changed the delimitation of various constituencies and created new demarcations - which basically was the biggest challenge - create a new map.
It looked like this (There were total 272 constituencies in total). Urdu team mentioned there were no geographical references to this newly drawn areas and ECP had put the new delimitations in the form of hand-drawn maps.
Josh was working on tilegrams so it looked like a good idea to make a new map using tilegrams, splitting and combining many constituencies. But tilegrams is a pain and we spent a few days on this but couldn't achieve what we wanted. Mainly because almost all the 272 constituencies had changes in some or the other way. Read more about tilegrams in Josh's post.
At this point, we thought to just include the table which was much easier and easy to navigate but it didn’t give the aerial view of who was leading and winning the election. We decided to give it a final shot and join all the provinces to at least form a image which has all new constituencies.
We were able to achieve the JPEG for the new map but we were yet to create a SVG and the shapefile. We took help from a friend of ours, who is an expert in Arcgis to look at the JPEG. He ran the JPEG on ArcGIS and georeferenced the outer boundary with the new map and later, cut and added the polygons to form the new map. After two days of craziness, we had the new map, which we later made it open source on Wikimedia commons under creative commons license for others to use.
The next challenge was to create a scraper which would pull election data from Election Commission of Pakistan’s website on the election day and feed it into our component. The problem? We had never seen how the live results page looked like and journalists in Islamabad told us it would be different from what is available right now for the 2013 results.
Scraping 2013 results also required some extra effort as ECP has redirects on the constituency result page (right now not available on the website). So everytime I made a request on the page I was redirected to the home page. I used Scrapy and made a request on the home page and then programmatically clicked on the individual links and then scraped the data.
Results are uploaded on the IDT dataset and a request is made from front end after every 5 minutes to fetch the new results. I wanted to get rid of the IDT this time because -
- IDT really struggles opening CSVs with >1000 rows and this one had >4000.
- During Karnataka elections IDT went under maintaince for a few minutes and I freaked out.
Joe suggested that I could uplaod the CSVs on S3 bucket and the uploading can also be automated. Boom! so easy, but unfortunately and thankfully(you'll know later why) we were short on time and couldn't test this and had drop the idea.
So we had this scraper (2013 results) and the scraper that was written for ECI and was used in Indian state results in the past. Hoping that the new page will be similar to before and one of them could be used with little changes made. Journalists in Islamabad were informed that it could take 1-2 hours till we see the new page and build a new scraper.
On election day
At around 8:30 PM we get a call from Urdu team that ECP isn’t releasing their election data in public domain. They had partnered with PTV network which were releasing numbers on the TV channel. The potential solution was to watch the TV and manually update the numbers (now you know why thankfully we did not go the S3 way. Journalists are familiar with IDT and they can do it on their own). But that sounded tougher than we thought. It was close to impossible to look at numbers and record it which were changing every minute.
But PTV was also updating numbers on their website and another scraper was written which would scrape data from PTV’s website. In another hour, we were able to pull data every 2 minutes which we would upload on IDT and publish. After publishing, map started to reflect the progressive results. The colours were in and we had a live map finally up on the website. (Yay!). Here we are celebrating with a pizza.
It worked like a charm until 3:00 am when PTV stopped feeding the results. The decision came after ECP stopped transmitting data electronically and decided to manually share the election data. At this point, the counting had slowed down. By 6:00 am we decided if the data doesn’t come in, Urdu team in Islamabad will have to update the IDT sheet manually and results would reflect on the map accordingly (again yay glad we did not go the S3 way). Here we are working at 3:00AM.
Becky in London started liaising with Zeeshan, Hassan and Abid from Urdu team and started updating the results manually. It went on for the entire day. After long 39 hours of manual updation, Election Commission finally updated the final results. Another scraper was written which pulled all the final data from ECP’s website and gave a final spreadsheet which was uploaded on the IDT and published to get the final data. And this is how it looked like.
You can see the live page here in case you missed it: https://www.bbc.com/urdu/pakistan-44922604
Analytics (based on Telescope reports from 25th to 30th July) - Total of 350,000 page views.
Things learned during this crazy process and things that could have been done better -
- If you're going to automate everything and eliminate the role of journalists then better have a backup plan which will be non developer friendly. IDT really helped the manual updating easy.
- Map could have a zoom option to focus on smaller constituencies. I did spend time on this but the outcome was not very smooth so had to get rid of it.
- Should have highlighted the selected constituency on the map.
- Should have stopped adding features/design changes a day or two before and started testing early. Could have this work on IE as well and maybe could have tried the S3 way too.