Fear of Photoshop - Layers and Masks

Feb 1, 2019 ~More adventures with layers! I promised we would learn to change a messy background and that is what we will tackle today. We are going to do a selection and change the background using layer masks. Photoshop has ten ways to do anything; you change your technique and approach based on your subjects and goals.

A bit of caution on making drastic changes to backgrounds. Choose your subjects wisely and perfect your technique or your efforts will be obvious and amateurish. Sometimes it is better to … adjust and blur the messy background instead of completely replacing it. If the light quality and direction on the subject is mismatched to the new background, even subconsciously the viewer will know it is faked. Of course, if you are doing Photo Art, then creativity rules.

Baltimore Oriole on a messy background

Here is our candidate. A lovely male Baltimore Oriole at Quintana during last migration. No sticks or twigs in the foreground but the background is too much in focus. And distracting with blurred sticks coming out of his head and tail. It happens a lot with little birds because the “background” is only a few inches behind the bird. Shooting wide open (f/4 or f/5.6) helps but if the bird turns you might have depth of field issues with part of the bird out of focus. If you are using a teleconverter to get more reach, then you might be shooting at f/8. Your depth of field is larger and those background sticks might be very sharp as well.

Silly bird, if he was on a solitary stick out in the open with nothing behind him for 10 feet this shot would have great bokeh.

Do you edits in ACR and open the image in Photoshop. We covered all of that in the first tutorial Overcoming the Fear of Photoshop. I save and work with .psd. but you might prefer .tif. Whatever you are comfortable with but not a jpg. You have your History and Layers panels open, right? Select the Quick Selection tool and click New Selection icon on the Options bar. Then click Select Subject. It is in a button in middle of the Options bar.

Options bar for Quick Selection tool. It look similar for any of the selection tools.

Options bar for Quick Selection tool. It look similar for any of the selection tools.

We did Select Subject in the last tutorial, but from the menu bar. After you click, the spiny thing spins and ….watch Photoshop do its $9.99 a month Majik. Then use the Quick Selection tool to add and subtract from the selection in the parts it misses, but this time, just get the main parts. If you need a refresher on selections, refer to Fear of Photoshop - Layers and Selections.

Then when you are done with the selection… we are not going to cut and paste like last time. Instead, we are going to do the refinements on a mask. Click on the Select and Mask button on the Options bar.

Select and Mask workspace is only available in Photoshop CC - this selection is shown on Onion Skin

Oh. Look at that. You are in a totally new work space. There is your selection with a slider panel on the right. It shows your selection and the mask around. Here is Photoshop’s article about Layer Masks and I didn’t find it really that helpful. But maybe you will. At least you can get familiar with the names of panels and processes.

Note the little Tool bar with 6 icons up on the far left and the familiar Options bar across the top. The slider panel is called the Properties panel.

Views you can choose to edit the selection and mask

Views you can choose to edit the selection and mask

Let’s get acquainted with this new work space. At the top of the Properties panel on the right, click the arrow next to the View thumbnail to see your choices. I am not sure what the default view for this is, but the image above shows your selection on Onion Skin. It might revert to the view you used last time. Press F to cycle through all of them; different subjects can be easier to work on depending on the view you choose.

You can even change the colors on the Overlay view. If you selected a bird on a serious green background, you might want to change the color to red for better visibility. Bill likes to check his on the Black & White view.

This work space is for making selections or refining selections you started in Photoshop with a mask on the parts you don’t want.

Make sure you uncheck Show Original at the top of the Properties panel to be able to edit this.

Tool bar

Tool bar

Quick Selection, Refine Edge, Brush, Lasso (with Polygon Lasso under) Hand and Zoom.

If you need to edit you can Add and Subtract from the selection or mask the same way we learned in the previous tutorial where we did a cut-and-paste to new layer technique. It is one of those mental concentrations we talked about last time.

Experiment changing the View to Overlay and using the Paint tool to edit. Add to the selection (the bird) and Subtract on background (the mask). Tip: Change the size of the brush as needed with your [ ] keys. Select the Subtract option and hold down Shift to toggle to Add on a PC. If you have selected Add, then hold down Alt to toggle to Subtract.

On a Mac, I think you use Option to toggle, but I couldn’t test it.

One of the extra cool tools here is the Refine Edge Brush tool. It is the second one on the little Tool bar. Select and click the Add icon on the Options bar and set the size at around 60. Zoom way in and carefully paint over the wispy feathers at the edge where the selection was not detailed enough.

Like these areas:

Paint out the last of the old background with the Refine Edge Brush around the wispy feathers.

Keep in mind how useful this technique would be for fur… or hair. It is useful to switch the views to check your progress. This tutorial is great at explaining the various tools and techniques: Changing the Background Using Select and Mask.

When you are pleased with your layer mask, set the Output To: New Layer with Layer Mask and click OK.

Layer showing image with mask (black) hiding everything except the subject

Back in Photoshop we have the selection on its own layer with a mask. Let’s look at that Layer panel. Layers are stacked like a deck of cards and you are looking down from the top to the bottom. Upper layers may block what is below, or individual layers may be complex.

Layer panel showing your new Layer Mask linked to a copy of the Background. The original Background is not visible

Layer panel showing your new Layer Mask linked to a copy of the Background. The original Background is not visible

The Background layer is turned off (no eyeball)

On the Background copy layer, we have a thumbnail showing the original, but locked to it is the mask we made. The mask is active (see the white outline around the mask thumbnail?)

Memorize this: “White reveals, black conceals”

If the black Mask with the bird shaped hole was placed on top of the Background copy, you would just see the bird. Which you do because the mask is linked to a copy of the background.

Since the black Mask is concealing the rest of the image.. we just see the transparency.

Make the Background copy thumbnail image active instead of the mask to see the effects. Turn the eyeballs on and off and observe.

Put it all back together like the image above and let’s go on.

New duplicate layer of Background - a copy of our original image with no mask applied is displayed.

Right-click on the Background layer and choose Duplicate Layer. A new layer shows up in the middle. Rename the new layer “Background to Blur”. This is like insurance; you are working in a copy. You can always delete it and start over.

Click the eyeball to turn off the layer with the Mask and the original Background.

Expanding the selection so it is not sharp

We are going to play around with blurring the background, but first we need to get rid of the bird on this copy. We don’t want bright orange blotches in the new background, and this way the bird doesn’t have to be exactly in the same spot as before.

From the menu bar Select > Subject. Let the program do the Majik selection.

When it is done, again from the menu bar Select > Modify > Expand. In the Expand Selection dialog, enter 10 or even 15 pixels.

Click OK. This will ensure when we do Content Aware you don’t have a sharp bird-shaped edge.

The $9.99 a month MAJIK

Make an outline around anything left out of your automatic selection with the Polygon Lasso tool. Just a rough selection will work.

Now… from the menu bar Edit > Fill and make sure the Fill dialog has Contents: Content-Aware selected from the drop-down list.

Click OK and watch the bird disappear.

You might want to select and repeat the Content Aware action on any really prominent parts of the background. We want just a hint of twigs and sticks in our new blurred background.

Bird is gone and ready to blur

Now. We are ready to blur the background. This is highly creative and subjective. A good place to start is with Filter > Blur > Gaussian Blur. Try differing amounts and then click the eyeball to display your bird (the Background copy with the mask) on the results. If you aren’t happy with it, change to the History panel and delete the blur action. The History panel shows the previous actions for each layer separately, so make sure you are on the right layer. Or you can use the menu Edit > Undo Gaussian Blur … or even Ctrl Z.

There are no limits. Experiment and try textures or other editing programs. Remember you can move your bird-with-mask layer; it doesn’t have to be in the same place. Use the eyeballs to preview new locations.

When you are happy with the results, then flatten your image. From the menu bar Layers > Flatten image. The program will ask “Do you want to discard hidden layers?” and that would be the old original background layer. Click Yes.

Baltimore Oriole on new blurred background

I used Topaz Glow (Fur and Feathers II at 20%) on the Background Layer to Blur. Then from the menu bar Filter > Blur > Gaussian Blur at 30 pix. Moved bird around a bit but it still didn’t look right. Then, I selected the Background Layer to Blur and from the menu bar Edit > Transform > Flip Horizontal. I like the flowers on the left, the shape of the background branches look better at this angle… and I also cloned in a bit more shading in the center.

It is a subtle change from the busy original but I think it improves the image. Little bitty flitty birds are almost always in the brush or dense foliage. There will naturally be twigs and sticks close by unless you are shooting at a drip or set up. But you don’t have to let distractions compete with attention for your subject.

Layer Masks are just one tool to combine with others in Photoshop and can be very useful in enhancing background. Sometimes waders are almost buried in reeds and ground cover. Even if you have a clear shot, the background can be too prominent and often these shots are unavoidable. There is a tricky way to handle this. Sometimes it works and others it doesn’t but you learn something every time you make an attempt. Each time you make a selection you perfect your Photoshop skills AND you learn more about the bird’s anatomy.

Clapper Rail with Layer mask and gradient

Layer panel for fancy gradient work

For this little guy I made a Layer mask to block the background but not the bird or foreground. You can see the shape of the mask on the Background copy layer in the Layer panel at the right.

Then I made a duplicate layer of the original background. Use the little icon with a circle in the middle at the bottom of the Layer panel to add a Vector mask to this layer. The program named it Background copy 2.

Then, on the original Background, Filter > Blur > Gaussian Blur from the menu bar. Slide the arrow all the way over to the right for 1000 pixels on the Guassian Blur dialog box.

Click the eyeballs so all three layers are visible.

At that point, select the Background copy 2, and make sure the white selection box is around the Vector mask. Select the Gradient tool and click about 1/3 the way down on your image, hold the left mouse button and drag down, release. You should see a blurring of the background. You can redo the gradient (click and drag) until you like the results; it starts over each time.

Last step is to activate the layer with the mask and blend the line with the Eraser tool set at about 50%. Flatten and save.

Do you think you have learned a little big about layers and backgrounds? Layers are really a YUGE part of Photoshop and I am learning all the time. I have another tutorial started on panoramas and I am thinking one on HDR techniques might be fun to do. What else? Let me know in the comments below if you have any suggestions. And please, if something is not clear or doesn’t work, let me know!

OMG... The Nikkor 500 f/5.6 PF VR lens is AWESOME

OMG... The Nikkor 500 f/5.6 PF VR lens is AWESOME

South to Surfside

South to Surfside