It's Not Just What You know, But Where You Look