Mapping Walkability – San Francisco
So for as long as they’ve been around, I’ve been a bit fan of Walk Score. It’s usually been just a toy, however as I get ready to move to San Francisco, I’ve been using it to compare against craigslist apartment rentals to the point where I think I’m getting RSI.
So it made sense to clear things up and take a look at the big picture
Read on for how it was done…
Very recently, they posted this amazing map of Seattle which blissfully shows the “big picture”. I was a little disappointed with the level of detail and legibility of the map: heat-maps are good for identifying trends and anomalies, but discrete levels can make it easier to read detail. When it comes to walk score, 2 or 3 blocks can change everything.
I got tired of waiting for them to release these maps for other cities, and decided to spare myself the back and forth between craigslist and walk score by rendering one of these overview maps for San Francisco. Mine certainly doesn’t have the same level of interactivity, but it is highly detailed (to the block).
I achieved this by a different kind of data mining than I’m used to. Usually I will write a small java program to download and scour an html file, follow links and mine for whatever it is I’m after. Map Score works only through client-side javascript and the google maps api.
So I wrote my scraper in javascript. Since Map Score uses the gmaps api, I know that whatever gmaps understands, Map Score will too. Turns out, you can give Map Score a latitude longitude pair. I tell it the top left corner and the bottom right, the level of detail I need, and I can loop through every lat/lon pair in the region. I call one page at a time, wait for the page to load, pull the number out of the page DOM and then load the next.
From here I use processing to create the map overlay. I use bicubic interpolation to determine the value of every pixel on the map, and snap this to my defined discrete levels of color. Snatch some screen grabs of MS maps and photoshop to take care of the rest.
I’m going to be referring to this map quite a bit as I get familiar with San Francisco and hunt for a new place to live. Hopefully this becomes a useful resource for others as well.
Update (July 14, 11:30 PM):
Thanks for the great response! Due to many emails asking for code or the ability to map their own metro, I’ve appended some source code to this post. Download the scraping code. Some description and instructions are included. The mapping code itself is a bit under wraps at the moment, sorry about that. But the magic is essentially Bicubic Interpolation.
Process is proudly powered by
5:16 am
Any chance of posting some code to get an idea of how you did the processing work?
10:40 am
Oh you.
I filter through my feeds like nothing else and everything you make always STOPS me and every time i catch myself surprised that it’s you.
11:36 am
you should figure out what is in the center of those dark spots that makes the walkability score so high! grocery store? mall? museum? MUNI/BART stops?
maybe get all the coordinates of safeways and MUNI stops in SF and put those on there, too….
i wish this kind of stuff had been around when i first moved to DC in 2003!
2:22 pm
This is a nice, well done project. The thing that strikes me, though, as an SF resident of 8 years, is that what you’ve done is identified where the flat places - both valleys, and high placed plateaus - are!
2:29 pm
Very nice. It does seem, usability-wise, that it’d help to have the colors spread a spectrum rather than be less transparent shades of the same color.
Who’s up for doing the plug-in/mash-up with all of Google Maps?
5:10 pm
Awesome. I thought about doing this once too, but you beat me to it. Care to share your code? I want to run it on my metro. Or would you rather just running it on a bunch of cities (including mine) and posting the results?
5:40 pm
Nice map, but I was disappointed to see my neighborhood (Ingleside) beneath the cutoff line. What gives?
6:06 pm
Thanks for the overwhelming response!
@Zach & Scott: I’ll try to collect bits of code I used to make this and post them here. At least the walk score miner.
@Greg: Sorry about the cutoff! This is the region I’m considering for an apartment. I’d love to map out more area, or more metros, but quite simply the data-mining process takes an eternity. This map uses a 75×50 grid of queries. If you figure each query takes 15 seconds, then mining the area took over 15 hours. I’m sure if I wanted to get fancy I could ad-hoc a distributed network to tackle it faster, but this was really just a weekend experiment.
7:39 pm
Nice idea, but it’s wrong about 20th street in Potrero Hill - there are many ups, downs, and flats on that street, yet it’s all coded pretty much the same color.
9:32 pm
I love this! I wonder if it takes into consideration the BART/MUNI stops. The red is certainly more saturated around them.
Congrats on the move. The SFBay is the best; I was discovering new things to get excited about daily for at least three months. Drop me an email if you have any questions. You know: where to get cheap Vietnamese tofu sandwiches or or where the best graffiti is, etc.
10:01 pm
@Erin: The map isn’t showing elevation or slope, rather it is showing a combined proximity to daily needs: groceries, restaurants, bars, parks and more. Check out Walk Score’s website for details.
Sarah posted a very interesting observation that areas with high Walk Score happen to usually be flat as well.
1:15 am
Retail tends to cluster much more intensely in a) flat areas and b) transit nodes such as subway stations, which explains the correlation between walk scores and proximity to BART as well as flatter sections of the city.
11:40 am
Could you please rename this map “Mapping Walkability - NORTHERN San Francisco”? You left out the entire southern 1/4 of the city.
12:51 pm
A question about your script: It seems that after the first point, it just keeps retrying the second point over and over again. Is this bit of script correct (line 152)?
if(retry!=null)
clearTimeout(retry);
retry = setTimeout(”tryAgain()”,Math.random()*60000+30000);
8:29 pm
Hi, the concentrations are concentrations of persons observed walking or? If so, how did you collect the data? Thanks!
4:46 am
@Pedro: I noticed that as well. I wasn’t sure if I was missing something. I find the scraper doesn’t work in Firefox 3, but had no problems in Safari. I found in Firefox it got stuck in an endless tryAgain(). I was going to look into it more today.
@Lee: Thanks a lot for making the code available. Really good stuff.
@Gisteacher: The data is taken from www.walkscore.com. From my understanding, they calculate the score on how many places/things/whatever are in walking distance. You can read more on their web site.
8:15 am
I was able to run the script and collect all the data. How can I turn them into a map?
5:22 pm
This is GREAT! I only wish–like another comment–that the colors were more differentiated.
Where I live is a 92 Walk Score, and a good reflection of the walkability. However, an important consideration for you is “What is the actual neighborhood like?” When it comes down to deciding where to live, the tool is a great starter point, but can’t substitute for walking around and getting to know what is actually within walking distance.
By the way, I learned about www.drivescore.com, where my home scores low. I wonder if it tends to be inversely proportional.
8:39 am
To everyone,
http://www.heatmapapi.com may be able to help you map, once you have the walkscore data.
12:10 pm
I enjoyed your map.
However i found a mistake.
You have wrong site information of Japan Town.
On your map, Japan Town is located between Gough & Larkin and Suuter and Geary.
Japan Town is located between Geary & Sutter and Laguna & Webster.
I should know as I live nearby.
8:12 am
Bardzo fajny Blog!
7:07 pm
Congrats for moving to SF! My favorite thing to do my first couple of years here was to hop a bus going to… who knew? and get off or on as my whims overtook me and walked around exploring the neighborhoods and then adventuring home in a similar way. It’s a great city!
1:01 pm
Lee–This is absolutely brillant! If only this were available on a google mashup for any area…
PS–Why didn’t walkscore do this themselves!?
5:43 pm
Hi Lee -
Any thoughts on where to get daily temperature data for San Francisco? One of the questions it’d be great to visualize in this way is what the San Fran micro-climates actually look like.
best regards, Mike
6:51 pm
do one for denver NOW!
7:57 pm
I am preparing a blog post for Discovery News about walkability. I’d like to post your map on the blog entry. Of course, I’ll credit the image. Please let me know if you’ll grant permission for me to use the image on the blog.
Thanks much,
Jessica
8:33 pm
Hi there! Found your site through Golan. Fantastic stuff, all of this! I want to echo Mike D’s request for a temperature map of San Francisco. You’ll understand this better once you’ve moved here, but the variety of microclimates is insane, and sometimes the temperature can drop 30 degrees within a few blocks, due to wind or fog. Where I live it’s so foggy in the summer, I have to look at a webcam from another neighborhood just to find out whether the sun is shining!
10:50 am
Interesting facts.I have bookmarked this site. stephanazs