Take the blue path behind the Inn (Caverne aux Brigands). The boulder is to the left of this path, past the wooden stairs, almost at the top of the hill.