I haven't done many updates in the last month, but that's because the site has reached mature point where I feel enough features are in place for it to run itself. With that being said, I was feeling another wave of motivation to make some improvements this week. One feature I've wanted to add for a little while was the ability to include road features such as bridges, gates, etc into the GPX download on the road page. I recognize not everyone wants waypoints included and so I've introduce a new download dialog that lets you select whether you want to include features or not. This work required a bit of an overhaul on how I store features in the backend, but now that's completed and in a much better place.

• Optimized the closure zone logic so that it's not re-closing the road every day.

Since adding in a basic map legend, many new features and map elements have been added to the map. None of these have been added to the map legend, as it was difficult to update. The design was also limiting and would be challenging to accommodate all the new map features. I've had it on my list to update for a while and now it's complete! See below or click the map legend icon on the map!

• Fix bug with closure zones where dates that wrapped around the year e.g. September 1st to January 1st were not working correctly
• Overhaul the user experience for adding bulletins and watching roads so that there are more clear calls to action
• Fixed bug where roads closed by a closure zone where not being updated on the home page

Users can now view pending, active and retired cut blocks on the map! Roads alone don't tell the whole story and if a given road has no reports, recent logging activity can give you clues about the status of the road. New cut blocks also indicate where logging road access is going to improve and what new areas will open up to explore. This update really starts to form the map into a cohesive road research tool. I hope everyone enjoys using the new overlays as much as I do!

• Updated the map layer selection menu and settings will now persist across page refreshes
• Updated the closure zone logic to support road exclusions
• Continuing to import or hand draw the wildlife vehicle closure zones
• Security and dependency updates behind the scenes
• Add better metrics around the registration flow

I designed the system from the get go to support importing new roads as the FTEN dataset (BC Gov's road data) evolved! When I first imported the road data, I kept track of a unique object (road) id from the government dataset, so that when I would reimport new data, I could check and deduplicate existing roads. In the name of moving quickly to develop features, I didn't do my homework and discovered a major issue with my approach. Yes.. this is a common pattern in my project, but so it goes in the pursuit of trying to deliver features as quick as possible.

Anyways, I found out the unique ID I was using changes any time a road is updated and as such not useful at all for eliminating duplicates. From this began a long journey of finding out that the BC FTEN dataset is managed by a lot of people, each with their own methods and processes and none of which are standardized. That means finding a set of properties to uniquely identify a road and track its evolution through the system is only possible in about 96% of cases. That's not bad, but 4% of ~269,000 roads is a lot roads to clean up and fix still. I even reached spoke with some of the GIS support team from FTEN and it turns out that most people simply import the latest version of the dataset, do their analysis and discard it. They usually do not need to persist road information and as such uniqueness of each road is not important. I also learned the reason that my existing road set has quite a few overlapping/duplicate roads. When a road is marked as retired, they generally stop displaying it on the map. So rather than modify retired roads, employees will simply draw over new roads and as such duplicates are born. I could resolve this issue for myself too by only displaying roads marked as active in the system. But I want to preserve all roads for historical reasons and because they may be of use for foot/bike travel still.

Through lots of trial and error I did find a somewhat palatable solution and had to adorn all existing roads in my system with a new set of fields from the dataset to make it possible. I couldn't and won't ever be able to eliminate all duplicates as it requires too many heuristics. But, I can get close enough to manage it for now. With that you'll notice that around ~3,000 new roads were added to the system today. The first of a likely bi-annual process to import the latest datasets.

I am considering one alternative, which is to take the latest road dataset and simply apply existing road reports/created roads on top of that. In that way, I won't have to do any deduplication checks. But it's risky if they remove old roads, or roads with reports associated to them. Something to think about for the future.

Other Updates:

• Updated closure zones to disappear when you zoom in to a road further.
  • This makes it easier to click around on roads and makes the map more legible overall
• Overhauled naming scheme of roads. In some cases they will be less meaningful names, but it reduces unwieldy and usually uninformative road names
• Uppercased road names when they're drawn on the map for better legibility
• Added location field on each road page
• Added report date field on the road bulletin feed to make it more clear when the report was for

An important aspect of any road system like this is being able to easily highlight key information to users. These are things like road closures, construction, gates and more that are important to know right up front. Prior to today, a big notice would appear on a given road page for any posted road closures. However, this system was limited as there are other types of notices that are worthy of being highlighted. What's more is that notices in most systems can become stale if they are not being actively cleaned up. That's a hard task in a system with 290,000+ roads.

That's why I spent a fair bit of time implementing a better notice system. Now admins can post important notices about anything that's worth highlighting and it will appear at the top of the given road page just like road closures. I've also added in an expiry mechanism so that notices can be given a specific end date and the system will automatically clean them up. I used this to my advantage to automatically post notices to roads where flooding was detected. Users viewing previously flooded roads will be made aware of the recent flooding and that road conditions may have changed. After 5 days the notice will expire and be removed!

Now I'm working on getting the system ready to import new road updates from the FTEN dataset. That's exposed a whole whack of issues that I'll dive into more on the next site update.

• Added in Texas Creek closure zone
• Fixed annoying popup bug on mobile devices where clicking a road would cause the map to move and the popup to disappear
• Fixed bug on iPhones where some road names would appear as phone numbers
• Improved the display of road aliases to make them easier to discover
• Added notification badge for mobile devices, if you have notifications
• Users can now change their location on their profile by editing their profile settings

These were just some minor updates while I work on overhauling the system that displays important notices about roads. I'm going to add support for temporary notices and displaying multiple notices at a time.

In order to protect wildlife during sensitive times, the government of BC often restricts motor vehicle access to roads running through certain areas for fixed periods. However, finding out about these closures requires you to consult this massive document and look for all roads you intend to cross. That's if you're even aware the document exists. These closure zones are poorly advertised and most people find out the hard way. Either driving for long distances to reach a locked gate or getting yelled at on social media after the fact for not knowing better.

Well the days of guessing are over! I've now added closure zones directly onto the map. You can see active closures as well as currently inactive ones to get an idea on whether your route will be impacted. Furthermore, if a zone becomes active all roads within that zone will automatically get marked as closed with a notice on the road page as well.

I'm starting with the closure zones in the Sea2Sky area first as those were available for download from here. However, the other zones need to be hand drawn and imported. I will do this in my spare time until they're all complete.

While working on a new feature to display wildlife closure zones, I had to reduce the number of waypoints used to draw these zones on the map. I was using a Kotlin version of Simplify JS and it occurred to me that maybe I could run this when returning road waypoints as well. It's actually not the first time I thought of using this, but my original idea was to modify all the underlying data to remove unnecessary waypoints. However, I never quite found it palatable to irreversibly modify all the road data like that. Simplifying the coordinates before returning the road waypoints to the map might be the best of both worlds here. It should in theory drastically reduce the amount of data transferred and time spent drawing the roads as well as leaving the underlying data untouched.

I had a feeling that it would incur a high server-side time penalty doing all that extra processing, but there's only one way to find out! So, I added in a test parameter to the api called tolerance that I could then set to different numeric values to find the balance between reduced road waypoints and a still well defined road. You see if you remove too many waypoints the roads will look like straight lines with sharp jagged corners. Not great for a map! The results were pretty spectacular. To my surprise the server side penalty was negligible. We're talking just a few ms!!

There was one problem though. If I reduced the number of way points by a certain amount, it would look fine at the given zoom level, but then as you zoomed in further the roads would look horrible. It then occurred to me that I could set the optimization level based on the zoom level of the map. When you're very zoomed out you can't see the finer details of the road anyways, so might as well aggressively reduced the number of waypoints. These zoomed out requests also happen to be the slowest as it's' including roads from a massive area. As you zoom in, you want more granular road data and so waypoints are reduced even less. Finally at the smaller zoom levels, the amount of data being returned is so minimal that no optimization is needed and we can return the highest fidelity road data. LeafletJS, the client side javascript map I use, actually already does this to save map drawing time. But the server still has to send all those unoptimized coordinates first. So it makes sense to do it on the server side too,

Results

To test the results, I chose 3 different zoom levels on the map and picked a relatively dense area. I then ran the requests multiples times for each zoom level to get a sensible average.

As you can see the response times and total data transferred are drastically reduced. This will make the map feel snappier than before, even with slower connections. The minor trade off is that some roads have a lower fidelity appearance at certain zoom levels, but its subjectively quite negligible.

Alright, this update is a subtle one, but in my opinion the numbers show it's worthwhile. It all started a few weeks ago when I discovered what each decimal place in the latitude/longitude waypoints of the road represented. I found out that in a pair of coordinates like 49.123456789, -121.123456789 the 5th decimal place represents 1.1m of precision and the 6th decimal place represents 1.1cm. Beyond the 6th decimal place the precision is so small that it's really no use for something like road waypoints.

Why is this relevant? Well the government dataset that the Service Road Atlas is based off uses coordinates with up to 13 decimal places!! Not only is it highly unlikely the government was measuring roads with nanometer precision, it's way more information than we'd ever need.

Let's take a road like Harrison East FSR. It has 2368 waypoints with a latitude/longitude having 13 decimals each. Remember, we only need 6 decimal places. So, that's 7 extra numbers for each part of the coordinate pair. Not only is that a lot of extra data to store in the database, it actually significantly increases the load times on the road map because all of that extra data which must be transferred to your browser.

I was aware of this issue a few months ago and put a small band-aid in place. Before serving the road waypoints, I would truncate the lat/long values to 5 decimal places which reduced the network transfer time. However, all that extra processing had a noticeable increase in CPU time taken to process the request before serving it. I knew I'd need to address it by updating all entries in the database to use 6 decimal places, but it's a bit scary to change the ~260,000 road entries in one shot.

With some testing, I decided that I wanted to use 6 decimal places instead of the 5 I had been serving. The reason was 1.1cm seemed better than 1.1m even if it's unlikely the dataset is really that accurate. The one downside to this is I'd be serving up slightly more data (there's one extra number in each coordinate now) but with less CPU time spent compressing/truncating the values. After some rigorous testing and backup policies in place I finally ran a coordinate optimization pass today and the results are instantly noticeable! The data transfer time for the extra decimal place is definitely offset by the reduction in CPU time spent serving up the request.

Testing was done against the same set of coordinates and the numbers are as follows:

So we saw ~7% increase in data transferred but that still resulted in 22% reduction in response time overall. Now for users with extra slow connections they may not realize this benefit, but I feel the average user will certainly feel the map being much zippier!

First update in the new year, as I was quite busy participating in interviews for a new role! I ended up getting an offer and so work on this site will slow a bit as I transition to a new company, but rest assured I will continue to develop some new ideas I have! The new updates this time around are minor but worthwhile changes and some bug fixes

• Added a contribution points system. You can view your points on your profile page and the top contributors over at: Top Contributors
• Fixed display bug where ampersands: & would get displayed as &
• Added support for link highlighting in comments and bulletins as well as newline formatting. This will give users more power to format the bulletins how they see fit
• Updated the profile page to match the bulletin feed design found throughout the site
• Fixed bug that would cause users to see a 429 error page when using the site as normal

Last big update of the year: "road watch" is now live. Registered users can now subscribe to road updates and receive notifications and emails whenever a road they're watching has a bulletin posted. I've also extended this system to forward comment notifications to your inbox if they go unread for more than 24 hours

Don't want to receive emails? No worries, there's an email preferences page where you can unsubscribe from your profile or through an unsubscribe link in your email.

Another big site update! You can now add replies to user contributed bullets! Have a clarifying question or just a general comment? You can now leave a reply and the creator of that road bulletin will get notified. This work required creating a notification system which paves the way for notifying users when roads change.

A few other changes of note:

• Fixed bug that would prevent users from seeing a useful login error if their email was pending verification
• Fixed bizarre bug where updating a bulletin photo caption with only one photo resulted in a 500 server error
• Increased session timeout to 1 hour (it was intended to be this way all along, but there was a bug)
• Added in most of the hydro projects around BC as these may indicate good road access

I will work on a road watch system next and email notifications (unless there's no desire for this)

Just a few small updates this time around, while I work on implementing comments/replies on bulletins

• Reduced icon sizes... these were like 2000x2000 px... whoops!
• Added clearer messaging about checking your spam folder after registration
• Updated/drew in a bunch of roads around the northern part of Harrison Lake
• Improved the legibility on street labels. The white outline too small and resulting in strange artifacts. This is now improved

• Improved SEO descriptions and OpenGraph images so that the site will be easier to find through Google
• Fixed 404 error handling for road pages

Facebook Login is now available through the login page! It has been my goal to have this feature enabled from the start, but other priorities took over. Many of the communities that might find this site useful are organized around Facebook Groups and so being able to seamlessly login and contribute has been a fundamental part of the experience I’ve been hoping to create. I originally created this site to improve on the existing services that are available. Many are behind a paywall or have a lacking user experience and I always felt that in order for a site like this to succeed, adding reports needs to be as frictionless as possible. The metrics and time will now tell whether this was a worthwhile addition, but if even 10% more users want to contribute via Facebook Login then that’s a success in my books!

Head over to the login page and try it out for yourself! Note that if you already have an account with the same email address as your Facebook, this login won't work. You'll need to continue to use your original email + password combination. For new users this should reduce any friction to sign up!

As for the technical details... because who doesn't love those? Well, it seems the golden era of Facebook API usage is over. Even Facebook Login which has traditionally been quite easy to setup, now requires a registered business and passing a vetting process in order to enable it. Thankfully, I was in a position to set this up, but I can see for most hobby style sites or personal projects it's a non-starter and that's a bummer.

The integration itself wasn't too challenging as Springboot does a lot of the heavy lifting for OAuth integrations. However, some thinking was required to figure out how to integrate Facebook users with the permissions system in use for locally registered users. I think I've found a tenable solution, although I'm not sure how idiomatic the approach is, but hey it's working. Time to focus on some other key features! A notification system has been asked a few times now as well as a means for editing photos on bulletins.

• Fixed a bug where zooming into certain parts of the map would cause the distance markers to reverse

• Well on my last trip we were trying to navigate along an old road and we inevitably lost it as it wasn't drawn on OSM or any other mapping applications.
• I've now added a "Download GPX" link to each road so that you can import it into your mapping app of choice! Note that you must be logged in to download the GPX file.

This was quite challenging and honestly the work is far from over. I've decided to release it in its current state to get feedback and to review the performance. I started out by using a Leaflet Plugin called StreetLabels. But it only worked well on very straight road segments and with numerous map breaking bugs. Adding to that FSRs have the unique property that they often have multiple steep curves and it makes rendering the road label quite difficult. To alleviate some of the problem I had to modify the plugin code quite heavily. I optimized the drawing of labels in a number of ways:

1. First, I added more padding around the labels to improve the collision boundary so that its less likely that they will overlap
2. Then, I wrote some custom logic to increase padding around the individual letters if the angle of letters changes more than a certain degree. This prevents issues where the letters would overlap on steep curves
3. The next issue was that each road segment had a huge amount of waypoints. This resulted in the letters being drawn in a jittery fashion up and down along the road. I fixed this by using simplifying or removing excess waypoints before drawing the line. Thankfully an existing solution existed, so I just needed to copy and paste from https://mourner.github.io/simplify-js/
4. As a final pass, I wanted to smooth out the path even further and reduce ultra sharp bends which would make the street labels illegible. For this I looked to a technique called spline interpolation. Again a nice algorithm in javascript was present already, so I needed only to apply it to the label drawing logic to make it work.
5. I briefly tested out an algorithm for culling road labels where the roads had too many extreme bends to feasibly draw it well, but it was not performant enough and too liberal in removing labels. I think a better approach will be to find the straightest segment of the road and draw the label there instead.

• Well, I'm finally back from vacation and dealing with jet lag, so the updates will trickle in slowly!
• Fixed a display issue with search results on the map where the text was not truncating correctly
• Updated/added roads around the Chilko Lake area and Lillooet. Many more to update still
• Added a privacy policy page here
• Starting work on login/sign up through Facebook. I have the login flow working and I'm just figuring out how to persist user registration info.
  • This will hopefully reduce friction to sign up and enable more users to contribute
• As a final aside, we've now hit over 260 road reports since the site launched in the beginning of October. This is really really great! Thank you to all who have contributed