Warning: readfile(http://alkoltashov.narod.ru/leebyron.com.txt) [function.readfile]: failed to open stream: HTTP request failed! HTTP/1.1 404 Not Found in /var/www/leebyron.com/www/how/wp-content/themes/leebyronhow/header.php on line 20

Mapping Walkability – San Francisco

So for as long as they’ve been around, I’ve been a bit fan of Walk Score. It’s usually been just a toy, however as I get ready to move to San Francisco, I’ve been using it to compare against craigslist apartment rentals to the point where I think I’m getting RSI.

So it made sense to clear things up and take a look at the big picture

San Francisco Walk Score Map

Read on for how it was done…

Very recently, they posted this amazing map of Seattle which blissfully shows the “big picture”. I was a little disappointed with the level of detail and legibility of the map: heat-maps are good for identifying trends and anomalies, but discrete levels can make it easier to read detail. When it comes to walk score, 2 or 3 blocks can change everything.

I got tired of waiting for them to release these maps for other cities, and decided to spare myself the back and forth between craigslist and walk score by rendering one of these overview maps for San Francisco. Mine certainly doesn’t have the same level of interactivity, but it is highly detailed (to the block).

I achieved this by a different kind of data mining than I’m used to. Usually I will write a small java program to download and scour an html file, follow links and mine for whatever it is I’m after. Map Score works only through client-side javascript and the google maps api.

So I wrote my scraper in javascript. Since Map Score uses the gmaps api, I know that whatever gmaps understands, Map Score will too. Turns out, you can give Map Score a latitude longitude pair. I tell it the top left corner and the bottom right, the level of detail I need, and I can loop through every lat/lon pair in the region. I call one page at a time, wait for the page to load, pull the number out of the page DOM and then load the next.

From here I use processing to create the map overlay. I use bicubic interpolation to determine the value of every pixel on the map, and snap this to my defined discrete levels of color. Snatch some screen grabs of MS maps and photoshop to take care of the rest.

I’m going to be referring to this map quite a bit as I get familiar with San Francisco and hunt for a new place to live. Hopefully this becomes a useful resource for others as well.

Update (July 14, 11:30 PM):
Thanks for the great response! Due to many emails asking for code or the ability to map their own metro, I’ve appended some source code to this post. Download the scraping code. Some description and instructions are included. The mapping code itself is a bit under wraps at the moment, sorry about that. But the magic is essentially Bicubic Interpolation.

Reader Replies

  1. Zach replied
    on Jul 14th, 2008 at 5:16 am

    Any chance of posting some code to get an idea of how you did the processing work?

  2. Jess replied
    on Jul 14th, 2008 at 10:40 am

    Oh you.

    I filter through my feeds like nothing else and everything you make always STOPS me and every time i catch myself surprised that it’s you.

  3. dan replied
    on Jul 14th, 2008 at 11:36 am

    you should figure out what is in the center of those dark spots that makes the walkability score so high! grocery store? mall? museum? MUNI/BART stops?

    maybe get all the coordinates of safeways and MUNI stops in SF and put those on there, too….

    i wish this kind of stuff had been around when i first moved to DC in 2003!

  4. sarah replied
    on Jul 14th, 2008 at 2:22 pm

    This is a nice, well done project. The thing that strikes me, though, as an SF resident of 8 years, is that what you’ve done is identified where the flat places – both valleys, and high placed plateaus – are!

  5. Dave Rutledge replied
    on Jul 14th, 2008 at 2:29 pm

    Very nice. It does seem, usability-wise, that it’d help to have the colors spread a spectrum rather than be less transparent shades of the same color.

    Who’s up for doing the plug-in/mash-up with all of Google Maps?

  6. Scott replied
    on Jul 14th, 2008 at 5:10 pm

    Awesome. I thought about doing this once too, but you beat me to it. Care to share your code? I want to run it on my metro. Or would you rather just running it on a bunch of cities (including mine) and posting the results?

  7. Greg replied
    on Jul 14th, 2008 at 5:40 pm

    Nice map, but I was disappointed to see my neighborhood (Ingleside) beneath the cutoff line. What gives?

  8. Lee Byron replied
    on Jul 14th, 2008 at 6:06 pm

    Thanks for the overwhelming response!

    @Zach & Scott: I’ll try to collect bits of code I used to make this and post them here. At least the walk score miner.

    @Greg: Sorry about the cutoff! This is the region I’m considering for an apartment. I’d love to map out more area, or more metros, but quite simply the data-mining process takes an eternity. This map uses a 75×50 grid of queries. If you figure each query takes 15 seconds, then mining the area took over 15 hours. I’m sure if I wanted to get fancy I could ad-hoc a distributed network to tackle it faster, but this was really just a weekend experiment.

  9. Erin replied
    on Jul 14th, 2008 at 7:39 pm

    Nice idea, but it’s wrong about 20th street in Potrero Hill – there are many ups, downs, and flats on that street, yet it’s all coded pretty much the same color.

  10. daretoeatapeach replied
    on Jul 14th, 2008 at 9:32 pm

    I love this! I wonder if it takes into consideration the BART/MUNI stops. The red is certainly more saturated around them.

    Congrats on the move. The SFBay is the best; I was discovering new things to get excited about daily for at least three months. Drop me an email if you have any questions. You know: where to get cheap Vietnamese tofu sandwiches or or where the best graffiti is, etc.

  11. Lee Byron replied
    on Jul 14th, 2008 at 10:01 pm

    @Erin: The map isn’t showing elevation or slope, rather it is showing a combined proximity to daily needs: groceries, restaurants, bars, parks and more. Check out Walk Score’s website for details.

    Sarah posted a very interesting observation that areas with high Walk Score happen to usually be flat as well.

  12. GG replied
    on Jul 15th, 2008 at 1:15 am

    Retail tends to cluster much more intensely in a) flat areas and b) transit nodes such as subway stations, which explains the correlation between walk scores and proximity to BART as well as flatter sections of the city.

  13. novabass replied
    on Jul 15th, 2008 at 11:40 am

    Could you please rename this map “Mapping Walkability – NORTHERN San Francisco”? You left out the entire southern 1/4 of the city.

  14. Pedro replied
    on Jul 15th, 2008 at 12:51 pm

    A question about your script: It seems that after the first point, it just keeps retrying the second point over and over again. Is this bit of script correct (line 152)?

    if(retry!=null)
    clearTimeout(retry);
    retry = setTimeout(“tryAgain()”,Math.random()*60000+30000);

  15. gisteacher replied
    on Jul 15th, 2008 at 8:29 pm

    Hi, the concentrations are concentrations of persons observed walking or? If so, how did you collect the data? Thanks!

  16. Zach replied
    on Jul 16th, 2008 at 4:46 am

    @Pedro: I noticed that as well. I wasn’t sure if I was missing something. I find the scraper doesn’t work in Firefox 3, but had no problems in Safari. I found in Firefox it got stuck in an endless tryAgain(). I was going to look into it more today.

    @Lee: Thanks a lot for making the code available. Really good stuff.

    @Gisteacher: The data is taken from http://www.walkscore.com. From my understanding, they calculate the score on how many places/things/whatever are in walking distance. You can read more on their web site.

  17. marco replied
    on Jul 16th, 2008 at 8:15 am

    I was able to run the script and collect all the data. How can I turn them into a map?

  18. Wako replied
    on Jul 16th, 2008 at 5:22 pm

    This is GREAT! I only wish–like another comment–that the colors were more differentiated.

    Where I live is a 92 Walk Score, and a good reflection of the walkability. However, an important consideration for you is “What is the actual neighborhood like?” When it comes down to deciding where to live, the tool is a great starter point, but can’t substitute for walking around and getting to know what is actually within walking distance.

    By the way, I learned about http://www.drivescore.com, where my home scores low. I wonder if it tends to be inversely proportional.

  19. Paul Irish replied
    on Jul 17th, 2008 at 8:39 am

    To everyone,

    http://www.heatmapapi.com may be able to help you map, once you have the walkscore data.

  20. Kazuko Tsuchiya replied
    on Jul 17th, 2008 at 12:10 pm

    I enjoyed your map.
    However i found a mistake.
    You have wrong site information of Japan Town.
    On your map, Japan Town is located between Gough & Larkin and Suuter and Geary.

    Japan Town is located between Geary & Sutter and Laguna & Webster.
    I should know as I live nearby.

  21. Alex replied
    on Jul 22nd, 2008 at 7:07 pm

    Congrats for moving to SF! My favorite thing to do my first couple of years here was to hop a bus going to… who knew? and get off or on as my whims overtook me and walked around exploring the neighborhoods and then adventuring home in a similar way. It’s a great city!

  22. Jason replied
    on Jul 24th, 2008 at 1:01 pm

    Lee–This is absolutely brillant! If only this were available on a google mashup for any area…

    PS–Why didn’t walkscore do this themselves!?

  23. Mike D. replied
    on Jul 24th, 2008 at 5:43 pm

    Hi Lee –

    Any thoughts on where to get daily temperature data for San Francisco? One of the questions it’d be great to visualize in this way is what the San Fran micro-climates actually look like.

    best regards, Mike

  24. do one replied
    on Jul 28th, 2008 at 6:51 pm

    do one for denver NOW!

  25. Jessica replied
    on Jul 30th, 2008 at 7:57 pm

    I am preparing a blog post for Discovery News about walkability. I’d like to post your map on the blog entry. Of course, I’ll credit the image. Please let me know if you’ll grant permission for me to use the image on the blog.

    Thanks much,
    Jessica

  26. Cassidy replied
    on Aug 30th, 2008 at 8:33 pm

    Hi there! Found your site through Golan. Fantastic stuff, all of this! I want to echo Mike D’s request for a temperature map of San Francisco. You’ll understand this better once you’ve moved here, but the variety of microclimates is insane, and sometimes the temperature can drop 30 degrees within a few blocks, due to wind or fog. Where I live it’s so foggy in the summer, I have to look at a webcam from another neighborhood just to find out whether the sun is shining!

  27. stephanazs replied
    on Sep 20th, 2008 at 10:50 am

    Interesting facts.I have bookmarked this site. stephanazs

  28. Evan replied
    on Oct 26th, 2009 at 5:46 pm

    Lee, can you (or anyone) post a version of the code you can confirm works on windows? FF, Chrome, Explorer all churn on the second pair of coordinates. Great stuff!

  29. Lee Byron replied
    on Oct 26th, 2009 at 11:52 pm

    Hey Evan, unfortunately I don’t have much time to revisit this code. It’s definitely not production code, I wrote it as a quick hack scraper to get the data I needed.

    It should probably run in Safari on Windows. The JS code isn’t all that complicated, so it can probably be tweaked to be more cross browser friendly by someone with the will to do so.

  30. Evan replied
    on Nov 10th, 2009 at 7:34 pm

    ha! Safari for Windows works. Brilliant!

  31. Soeren replied
    on Feb 3rd, 2012 at 3:02 pm

    Is this script still working? I get an error message that walkscore couldn’t find that location while running the script.

Leave a Reply