The force of gravity exists where there is mass. It is not a very strong force, which may surprise you ... until you consider that it takes the entire mass of the Earth to hold you to its surface, but you can break contact easily by jumping.
Smaller masses cause less gravity. If you were standing on the surface of the moon, you would experience a force of gravity only one sixth of what it is on Earth. You would weigh only a sixth of your Earth-normal weight.
If you were standing on tiny Eros, which is an asteroid a mere 34 kilometres long, the force of gravity would only be one thousandth of Earth's gravity ... you would easily be able to jump into space. But this gravity is still enough to make you feel 'down' when standing on it, and to make rocks on the surface roll downhill.
Strange effects like this can even be observed on Earth. We usually consider Earth to be a perfect sphere, but it isn't, really. It's slightly wider at the equator, and its surface is covered with mountains and valleys. (It very nearly is a perfect circle anyway ... draw a large circle on a piece of paper and the real bumpy surface of the Earth will be hidden within the thickness of your pencil line).
Every bump that is a mountain has mass. Moreover, every large mineral deposit under the surface has more mass than if it were just rock. All of these masses also cause a force of gravity.
The result is that when you are standing near a mountain, or an underground mineral deposit, the force of gravity from these large masses adds its pull to the Earth's, causing the net force to be slightly at an angle to 'down'. Or to say it differently, 'down' is now slightly off vertical! Objects don't fall exactly towards the centre of the Earth, but at a slight angle to the vertical, as they are pulled sideways.
Mountains and mineral deposits are smaller even than Eros, and so their force of gravity is very tiny compared to the gravity of Earth. Consequently the angle at which you stand, and at which things fall, when you're near a mountain or mineral deposit, is too tiny for you to notice.
But this angle is detectable by sensitive instruments. Mineral deposits actually can and have been discovered by looking for variations in the force of gravity from place to place.
In the late 1960's, the first satellites to orbit the moon in preparation for manned landings always seemed to veer off their predicted course ... until scientists realized that mineral deposits under the surface of the moon were causing fluctuations in gravity from place to place, and resulting in unpredicted changes to the satellites' orbits. These 'mascons', or mass concentrations, under the surface of the moon, were later mapped by using the changes in the satellites' orbits.
Things get even more interesting when we simulate gravity by rotating a space station. Centripetal force caused this way imitates gravitational force. On the inside, the rotation causes a person to move outwards, and the floor provides the equal and opposite force. This makes 'down' perpendicular to the floor, wherever you are on the inner floor. A large enough cylinder constructed this way and caused to spin could allow everyone inside to experience simulated gravity, rather than the usual weightlessness of space. People living and working on the inner side will look up and see others above them, upside down, walking around and working comfortably.
In this situation, 'down' is always 'outwards towards your feet'.