Think of your film as a roller coaster. Have you ever noticed that the ride up on the roller coaster is the part that has people most scared? Some fear the drop, and become nervous because dropping down is now inevitable. Some people begin to think about all of the things that could go wrong. The roller coaster could fall apart. Everyone could die. Perhaps you fly out of the coaster. Once you go down that hill, there's no more fear and you know that you're okay. The next hill isn't as scary for people because you've already been on one, and you'll know you'll be okay. If you throw people in another direction, which makes them fearful of what could happen next. Now that you've thrown them off, they don't know what to expect.
Create a realistic environment with characters the audience cares about. When the audience cares about a character, they will be fearful when you put that character into danger. When the audience thinks that this person that they care about is in danger, they begin to get nervous. Create complex and interesting characters that people can relate to through good writing and acting.
Once you've created relatable characters and a realistic world for these to live in, take the characters up the roller coaster and towards the drop. The imaginations of the audience go wild. They think about everything that could go wrong.
Now, thrown your audience off. Whatever route they think they were going, throw them off, and start going an unpredictable direction. A huge problem with most horror films is that they are usually predictable. People who have been on coasters before think they know where they are going. Thrown them off, and their minds will begin to go crazy with ideas. They don't know where they are going and what could happen to them.
----------
Finding ways to create genuine fear is difficult though. You have to find things that will scare people, and use those as the basis for the ride you are taking the audience on. You have to find a fear that you understand and know how to toy with. You may want to study up on that fear to have more knowledge of how that fear is triggered. Know how to evoke that fear and how to translate that fear into the form of a film. Sometimes playing off of current fears might scare people. If there is fear of terrorism or war, then making a post-apocalypic film will probably evoke strong fear from an audience if done well. If there is a new disease or growth of how common a disease is, then making a film about some sort of sickness will scare people.
Understanding fear and the human mind helps as well. Wes Craven, considered one of the best horror directors of all time, studied psychology. He knew how to toy with the minds of audiences. Not to mention that around the time that
Nightmare on Elm Street was made, there were numerous deaths, the most possible cause of death being scared to death by nightmares.
There are many techniques used in horror films which can be altered and used in different ways to suit your film:
Sound
There are many things that can be done with sound. For example, if we hear distant footsteps that begin to get louder and louder, we are going to be afraid. If you just thrown in some sound effects of footsteps, there's no fear. The illusion of broken because the sound of the footsteps is unrealistic. Sound is another thing that can get the imagination of the audience going. In
Texas Chainsaw Massacre, various sounds were used to tell the audience that someone's body was being mutilated. The sounds of crunching bones and ripping flesh sounded horrible. It was cringe worthy. Without those sounds, the film would have been far less effective. The skillful use of Foley and sound effects made the film much more intense than it would have been without them. Sound can be one of the most effective elements in film.
In real life, there is ALWAYS sound. By eliminating all sound, it can be unnerving and seem odd, which if that is what you are going for, it can be very effective.
Music is important as well. In the film
Jaws, music was used to let the audience know something was about to happen. The shark didn't immediately jump up and kill people, Spielberg teased the audience. He didn't show the shark with blood and nasty sound effects, he built up to the gory climax of the film with intense music. Music can also be a tool for shock moments. In the opening of
Insidious, the sight of a darkly lit stranger peering into a house paired with loud screeching strings make for an unsettling moment. The lack of music can be frightening as well. In Ryan Connolly's
Tell, there was a moment when we are leading up to what could possibly be a scary moment, then cut the music out. We are now alone with the character, following him into darkness. There's no longer music to support us or keep us company, we're now all alone. Other examples of lack of music being scary would be
The Shining and
The Ring. Music, if done right, can be absolutely chilling and set the tone of a film.
Halloween,
Suspiria, and
Profondo Rosso were greatly assisted by their chilling scores and wouldn't be the same movie they are without them.
Lighting
Lighting is very important. Darkness can again trigger the imagination of audiences because there is something hidden in it. Sudden flashes of light can be shocking. Under lighting can make us fear a character. Light can draw attention to certain parts of the frame, and hide other parts. Light can mislead us. Light can reveal things to us. Brighter and more stylized lighting can work as well. Check out the work of Dario Argento or at least some clips of his films to get what I mean. Light can also hide objects and people from us.
Camera Work
There are many different shots that can evoke fear in audiences and work in horror films. Wide shots can give a feeling of being alone, closeups can make us feel closer to a character in a situation. POV shots can make us feel more involved in a film and feel the fear that a character feels. Shots that look down upon a character can make a character feel smaller, and shots from above can make a villain feel bigger. Dutch shots or uneven shots can give a feeling of something being wrong. Handheld can evoke a feeling of unrest. OTS shots or shots were an object is blocking part of the frame can make it seem as if a character is trapped. Shallow DOF can make an area seem smaller make the audience feel more claustrophobic. Dolly shots can pull us into a location that we don't want to be in. There are many, many, many shots that can make horror films scary.
Editing
Editing often determines timing. Lingering on one shot of a door or an object can make the audience feel uncomfortable. A couple of frames can make a rubber monster look real or fake. The editor of Jaws said that if the shot of the shark jumping onto the boat near the end of the film had been 10 frames longer, it would have ruined the illusion. I could go on and on about the power of editing and horror, but if you do attempt to make a horror film, you'll realize that editing is incredibly important.
Story
Story is very important. In order to really scare people, you have to have a good story. If you have plot holes, major continuity errors, or just elements that make no sense at all, then it's going to fall apart and ruin the whole experience (unless of course you are doing an experimental piece, ex. Eraserhead)
Pacing and Timing
Pacing and timing are SOOOOOOO important. Pacing and timing isn't just editing, it's the soundtrack, the story, everything. For example, in Friday the 13th, the ending scene went on so long that people didn't think that anything would happen. Right when people got comfortable, they threw something at us. Pacing shock moments and scenes of suspense are important as well.
SFX & VFX
There's not much to be said about these two. If they don't look realistic, the illusion is broken.
---------
Some things that I find most effective are misleading the audience, unpredictability, and making the audience afraid of nothing. In the new film,
The Conjuring, there are many scenes when we thought we knew what would happen, then they threw something else at us that we didn't see coming. There were moments when the audience thought something was going to pop out of this closet, then something pops out of nowhere from somewhere else. That to me, is scary. The use of lighting and sound to mislead the audience make the scary moments scarier then they would have been. In
Alien, there is a moment when a character is crawling through vents, and the alien pops up when we least expect it. We're just figuring out where we think the alien is, then it just pops up. The whole audience screamed. In
Cat People, there was a scene when a character is running from what she things is footsteps behind her, then a bus pulls up, making a loud noise. There was nothing scary in the scene. The imagination of the audience made the scene scarier than it really was.
-------
I recommend you watch and download
Tell by Ryan Connolly. He has a director's commentary where he explains all of the techniques he used in the film. It's both enjoyable and informative.
http://triune-store.myshopify.com/collections/films/products/tell-hd-digital-copy
Here are some articles/pages/videos to check out:
http://www.borgus.com/hitch/
http://www.tested.com/art/movies/455206-rules-terror-horror-movies/
http://vimeo.com/videoschool/lesson/147/scary-shooting-techniques
http://www.pandolin.com/techniques-to-shoot-a-horror-sequence/
Good luck!