IMHO the best horror stories are the ones that don't tell the whole story -- there must be a possibilty of something diabolic at work, but not the clear certainty. There should also be a clash between modernity and the primitive. Consider the classic ending of chapter 2 of the Hound of the Baskervilles. Dr. Mortimer, whose plot role is largely expository, has just described the legend of a hound which haunts the Baskerville family across the generations; Holmes dismisses it as a fairy tale. Mortimer asks Holmes to bear with him, as he describes the death scene of the most recent Baskerville, who, based on the facts that were presented at a public inquest, appears to have perished from a heart attack while waiting for someone outside of the family's ancestral manse.
But Mortimer goes further and intrigues Holmes deeply when he tells him that there were certain facts that were not presented at the public inquest, and these were that Mortimer himself had seen footprints near the body of the deceased Baskerville. Holme's then asks if they were a man's or a woman's. Mortimer then gives the great classic horror line:
"Mr. Holmes, they were the footprints of a gigantic hound!"
The rest of the novel is devoted to explaining this observation as the modernity of 1901, a world not so different from our own, struggles with the concept of an ancient legend apparently come to life. Starting in 1914, the story has been produced in film (and later on television) on numerous occasions.
Set in almost the exact same time (Valentine's Day, 1900) was Joan Lindsey's 1967 novel Picnic at Hanging Rock, concerning the (fictional) unexplained disappearance of several school girls at a real place, Hanging Rock, in Victoria, Australia.
Peter Weir, the Australian filmmaker, produced a bewitching film version of the story in 1976. Both the novel and the film present the circumstances as real, or at least not clearly fictional, and I recall as a young man seeing the film and then trying, unsuccessfully to look up the real instance in the library (in those pre-internet days, a "library" was building with a lot of books in it).
Related to these fictional events are the popular imagination of several actual unexplained disappearances, notably the brigantine Mary Celeste, whose crew had mysteriously abandoned the still-seaworthy vessel in 1872, the incident in 1900 of the missing Flannan Isle lighthouse keepers, and of course the 1945 incident of the five missing planes and 14 crew, of Flight 19 flying out of NAS Fort Lauderdale.
Regardless of rather conclusive evidence as to the likely causes of all of these disappearances, the idea of of weird supernatural intervention has taken root for all these. The settings of each instance (ship, lighthouse, aircraft) were isolated from civilization and exposed to nature. Both the 1872 Mary Celeste and the 1900 Flannan Isles incidents occurred in the Victorian era, a time of enormous scientific and technological change, proceeding as fast or faster than our own, which enhanced the public impression of mastery of the natural world, and the clear shock when such proved (of course) not to be the case. Similar attitudes and reactions permeate the Flight 19 incident in the public mind.
So, if you want to tell a really scary story, you need to leave out a complete explanation. Set up the circumstances and let the public frighten themselves. Great horror films, such as Alien follow this principle. Also, in the original Halloween, Michael Myers somehow disappears when he should be dead. Why he can and where he goes -- these are unanswered questions, left for the audience to imagine. Recently, in the Walking Dead, the plot led up to a clear understanding of something horrible being kept in a bag, but it's quite properly never shown -- far more effective.
In 2001: A Space Odyssey, there were a couple of plot choices that were resolved quite effectively. Apparently the black monolith which sparks the transformation of man from animal to human was originally going to be a Lucite slab which showed images to the apes, sort of like Army training films or the Far Side: ("Og smack Thag with bone"). Yikes. Fortunately the Lucite slab idea didn't work out and they when to the much more effective black monolith theme.
Then my understanding was that HAL the computer with human emotions, was going to be a mobile Robbie the Robot type of character, but Arthur C. Clark rejected this idea on the grounds that the scientific trend would have been away from such things, and so the discorporeal HAL -- and much more effective and scary -- came to be. Similarly, in the very good Disney production from the 1950s, Forbidden Planet, one never sees the monster, only its effects. Prometheus (2012) tried to cover similar ground, but made the error (among other mistakes) of attempting to depict the God-like aliens explicitly when they should have been left to the imagination of the viewer.
Most people have a rational self which tells them that there are no demons, devils, gods, and so forth. But people can scare themselves quite well -- this is the cause of Salem Witch Trials and much other human misery. The very scary (and much imitated) Blair Witch Project takes full advantage of this, allowing the characters (and the audience) to be terrorized -- maybe by the supernatural -- but certainly to a large part by themselves. BWP also follows the rule of never showing the (possibly) supernatural antagonist.
One BWP imitator was the Chernobyl Diaries (2012) which seems to start out on a promising idea: callow American youths get their comeuppance when they mess around with illegally tourist a radioactive ghost town in a foreign country. Unfortunately the movie seems to know but nevertheless violates the rule of not showing the antagonist. Found footage (the premise of BWP) is used in part of the film and rather effectively at that -- this method creates the impression that the horror is not externally caused but rather contributes to the impression that the characters are creating the fear themselves. But ultimately and disappointingly CD becomes a rather poorly patched together set of cliche horror scenes.
One effective method of horror is to place the characters in an escape-proof situation (spaceship, creepy mansion, etc.) and then bump off them of one at a time, a technique made famous by And Then There Were None (1945); for this to work, there has to be good character development -- the audience must be able to readily distinguish the characters, and have some reason to care about their fate.
This places a practical limit on the number of potential victims. ATTWN had eight, and Night of the Living Dead, Alien and Halloween all had seven, Deliverance (1972) had four and Straw Dogs (1971), a sort of Yokel Apocalypse movie, had three.
Prometheus sought to use the proven ATTWN-Alien steady attrition method, but started off with 17 potential victims -- the audience was lost already from the start. Stalingrad (1993), where war takes the place of the supernatural, had 20 or more potential victims, none of whom could be told apart, at least by me anyway. Chernobyl Diaries starts off with the right number of people (seven) but six of them play fungible (and boring) American twenty-somethings.
Finally, make sure the characters can't reach a place of undoubted safety. NotLD was excellent in this regard, but so also was War of the Worlds (2005), in particular the ferry scene. World War Z seemed to be aware of this rule, but didn't follow it to the logical conclusion.
So, to recap, the rules for successful horror would seem to be:
1. Keep the number of potential victims down.
2. Develop the characters a bit so the audience cares abou their fate.
3. Don't show (or at least show very little of) the threat.
4. Place the characters in situations where they can't get outside help.
5. Leave doubt in the audience's minds as to whether any refuge is in fact safe.
6. Always leave room for a logical scientific explanation for the events.
7. Build up a situation where modern culture cannot seem to prevent the occurence.
8. Permit an inference that the horror is within the minds of the characters.