- June 9th: Django exploration
- June 10th: Decide on basic design plan
- June 18th: Finish initial implementation of Tor Weather in Django (old functionality only)
- June 25th: Finish adding new features to Tor Weather
Today we discussed several issues involving the future of Tor Weather, including if/when to email node operators in various scenarios (i.e. if they are running an out-of-date or obsolete version of Tor). If these clients haven't signed up for Tor Weather, it is possible to parse their email from an unformatted contact field listed in the directory documents that are continuously updated by the Tor server. We hope to add functionality to Tor Weather involving an options menu for clients that would allow them to opt into or out of certain types of notifications (i.e. if their bandwidth has reached a critical low or if their node has earned them a T-shirt).
Met with Andrew Lewman from the Tor Project and discussed the above issues regarding Tor Weather, as well as various topics regarding Tor Status. Began work on an email address parser for Tor Weather.
We decided to rewrite TorWeather from scratch using Django. Today we went through the Tor tutorial and discussed design options for possible models in the new framework. We posted our tentative plans for the model design on the Tor Weather page. We ended the day with a bit of coding and attempted to better understand git.
(Timeline: Decide on basic design plan)
Completed database design. Began rewriting Tor Weather in Django. Explored TorCtl code to figure out how to use it.
Today we continued writing Tor Weather in Django. We discussed MVC design as it relates to where and how certain functions should be handled in our code. Coding for the controller is underway, as are the HTML templates for the different pages users will see.
Today we added more HTML templates, including an error page with useful information when a node fingerprint isn't found, a change preferences page, and a preferences confirmed page. We added relevant controllers for these pages as well. We decided that the flow for sending reports to users should be to iterate over each router in the database, attempt to collect descriptor and consensus information for that router, and update all subscriptions linked to that router sending an email if necessary. Initially we were planning to iterate over all of the subscriptions in the database, but since there may be several subscriptions per router, this method seemed cleaner to us. We're still unclear about hibernating nodes and decided to look into the specifics of node hibernation tomorrow (i.e. what it means when a node is hibernating and how a hibernating node is functionally different from a node that is down).
Today, we continued work on HTML templates, added functionality for updating the global router database, and added functions for adding various entries to the database. We discussed in length how to deal with hibernating nodes, and decided that if there doesn't exist a consensus document for a node but there does exist a descriptor file with the 'hibernating' flag, we will not email the node's subscribers.
Today, we added a lot of small things to fill in the gaps in our current design. We hope to have a functioning version of Weather up to test tomorrow.
Today we deployed a local version of our Tor Weather implementation and played around with it to see where the bugs were. Turns out there were many, so we spent a lot of time fixing syntax errors and debugging.
(Timeline: Finish initial implementation of Tor Weather in Django (old functionality only))
Got the prototype Tor Weather implementation to run successfully. Can handle new subscribing and unsubscribing (using a gmail account to send emails). Getting consensus documents and running the updater upon consensus document arrival is still not working, though. Worked on lots of debugging, and also started on the interface for the subscription page that will have the full list of features.
Today, we changed the database structure to make it more object-oriented. Before, there were fields in the Subscription model that were only required for certain subscription types. Now, each subscription type has a separate model that inherits from the Subscription model. Additionally, we can now populate the Router table in the database from a consensus document. We're continuing optimization of our design in general, and we're starting to play around with expanding the Tor Weather functionality (i.e. adding new subscription types, like T-shirt notifications).
Today, we got Weather to automatically update the global router database and send appropriate notifications when a new network status event is received. We also fixed more bugs and email formatting issues. Additionally, we have begun work on Django templates for a web interface that includes the new features we are planning on implementing.
Today we got the base Tor Weather functionality working to send node-down notifications. We came across new error types that should redirect to error pages (i.e. if a user tries to confirm their subscription twice or if a user tries to change their preferences before confirming their subscription), and we implemented these pages. So that controllers don't wait for emails to be sent in order to return and display pages, sending emails is now handled by daemon threads. We also prepared for tomorrow's presentation, which took up a good chunk of the afternoon.
Today, we made an initial implementation of the welcoming system that sends emails informing new relay operators about the existence of Tor Weather. This included writing an email parser to get emails from the contact lines of descriptor files. After further testing of the web pages, we fixed a few minor issues and added some new functionality. Now, a user can request to have their confirmation email resent to them if they happen to lose it. There are new error pages (404, 500, a page for if the user tries to confirm their subscription twice and a page if the user tries to change their preferences before confirming their subscription). We're continuing work on a prototype page for the new subscribe form with the full feature list.
Today, we made a T-shirt subscription type and implemented checking if a user has earned a T-shirt. We added functionality to send a welcome email to all stable node operators (not just those who've signed up for Tor Weather) and got the email parser working successfully. We're making progress on the new subscribe page as well.
Today we implemented the bandwidth notification system and started work on our final notification type, the version notifications. We created standalone documentation for the entire Tor Weather application and reconfigured the emailing to collect all of the emails and send them en masse instead of reconnecting to the server to send each individual email.
Today we began testing, fixed various bugs, implemented the version notification system, and continued work on front end design and forms submissions.
Today we wrote tests for subscription requests, fixed bugs, and continued work on the front end forms. We also found a more intelligent way to calculate a relay's bandwidth and implemented it.
Today we mostly fixed bugs. We also finished the new subscribe page. In addition, we began learning how to effectively use Prof. Danner's private Tor network script to fully test Tor Weather's notifications.
We're starting to look into our next project, Tor Status. The existing Tor Status is essentially a bunch of PHP files, and the Tor staff wants us to rewrite the Status application in JSP/servlets as an extension of the Tor Metrics application, which collects data about the Tor network and displays relevant graphs on the Metrics web page. We've started looking through the source code for both Metrics and Status, and we're starting to learn the basics of servlets and JSP. We're putting the finishing touches on Tor Weather, including updating the preferences page to include the necessary fields for the new notification types.
Today we thought more about what to do about Tor Status, mainly just waiting until tomorrow to see if Andrew and Karsten debate between themselves any more. We also finished implementing the preferences page, and worked more on the virtual Tor network.
We finally got the private Tor network working, which will allow us to test the Tor Weather application much more rigorously (we've already come across several bugs). In talking with the Tor developers, we've gotten a clearer picture on how to move forward with Tor Status as well as some starter code to work from. Otherwise, we're putting the finishing touches on Tor Weather and hope to see it finished soon.
Today we began testing the notification types (specifically node-down subscriptions) and welcome emails using the testing Tor network. We came across a few bugs, but so far the notifications seem to be sending correctly. We improved the email parser to retrieve more emails from operators' contact fields.
Today we continued testing our Tor Weather application using the testing network, which has been helpful in catching a lot of bugs and other issues. Professor Danner gave us a tutorial on JSP, which will help us as we launch into Tor Status in the next few days.
Today we finished testing the Tor Weather notification types. We got Tomcat up and running and started learning JSP/servlets.
Today we made improvements to our code based on comments in our code review, including cleaning up our documentation. We're currently trying to determine why our test code is failing and hope to have the issue resolved tomorrow. We're also troubleshooting auto-complete functionality for the fingerprint field, whereby the user can type either the name of their node or the beginning of the fingerprint and be supplied with possible matches.
Today we worked with Professor Danner to get a model JSP web application to work. We're starting to play around with it to get a better understanding of how JSPs work in order to start development on Tor Status.
Today we discussed what features the new Tor Status should implement and how best to display them. For the time being, we decided that the main Status page will display a single graph of the number of routers in the network over time. The page will include a search bar to find a particular router, will allow the user to list all of the routers in the network with some basic statistics, and will link to pages displaying network-wide (rather than router-specific) data, such as the the number of routers by country and a histogram displaying the number of routers with a given uptime.
Today we started using postgreSQL to play around with the database we were given by one of the Tor developers. We're finishing up documentation on Tor Weather, which is (presumably) the final hurdle before we send it off to Tor.
Today we learned that the Tor servers that will be used to deploy Tor Weather are running an earlier version of Django. We started backporting to the earlier version today. The process will likely take a while, since the versions (1.1 vs 1.2) are significantly different. We also altered the conditions for sending welcome messages. We decided to implement a two-day startup period following the initial deployment. Any routers seen during those first 48 hours will not be sent a welcome email. That way, we hope only to welcome new stable relay operators.
Today we finished the backport of Tor Weather. We also fixed some small bugs and artifacts that were discovered in the process.
Today we got database access working using PostgreSQL. We can now successfully view the JSP pages that Karsten, one of our project mentors, sent us. We plan to create a Database Access Object to provide a layer of abstraction between the application and the database. We anticipate that Tor Weather will be deployed fairly soon, once we hear back from our project mentors.
Today we started looking at open-source Java graphing utilities for Tor Status. We started writing a servlet to generate a graph of bandwidth over time using JCharts.
Today we decided to use JFreeChart (another charting utility) instead of JCharts. We started learning how to create graphs using JFreeChart and incorporate it into the Status application. We worked on the web front end and got it to display all of the routers "in the network" (via our database dump) cleanly.
Today we began refactoring several methods into models in order to simplify the updater module and untangle the subscription-specific code from any modules other than models. This increases ease of development in the future, because now the code only needs to be modified in models and not in emails and updaters.