Exfoliating — using a brush, scrub or topical treatment to remove dead, dull cells from the surface of skin — can feel like a magic trick. “It gives an immediate brightening effect to the face,” says Mary P. Lupo, MD, FAAD, a clinical professor of dermatology at Tulane University School of Medicine in New Orleans.

Exfoliating also encourages collagen production, helps make fine lines and wrinkles less noticeable and allows serums and other treatments to penetrate more deeply for better results.

Not everyone benefits from exfoliating, though. It can worsen some skin conditions. And even people with normal skin should be careful when they exfoliate.

Related: Are You Sacrificing Health in the Name of Beauty?

The dark side of skin brightening

Exfoliating, particularly when it involves harsh scrubbing, can cause some people to develop post-inflammatory hyperpigmentation (PIH), a condition in which dark spots form on skin. Especially at risk are those with dark skin or who tend to get dark spots after bug bites or burns.

Exfoliating also can worsen certain skin conditions, says Lupo. For people with inflammatory acne, aggressive exfoliation “can de-roof a lesion, which can increase the risk of infection, cause a scar or both,” she explains.

Safer alternatives if you have acne: Rebecca Baxt, MD, a board-certified dermatologist in Paramus, New Jersey, recommends using a clay mask or a chemical treatment to brighten up acne-prone skin.

Look for peels made from alpha hydroxy acids or AHAs (glycolic acid, mandelic acid and fruit acids), beta hydroxy acid or BHAs (salicylic acid and willow bark) and polyhydroxy acids or PHAs (gluconolactone and lactobionic acid). Lupo recommends over-the-counter products with low acid concentrations — no higher than two percent salicylic acid and 10 percent glycolic acid.

Retinol (vitamin A) doesn’t exfoliate directly but it’s a great complement ingredient, because it stimulates cell turnover, adding “in with the new” to your “out with the old.”

Gritty scrubs also can cause flare-ups of rosacea (a condition that causes redness and bumps) and eczema — worsening redness and irritation of skin that’s already sensitive. If there are pimples with the rosacea, there’s also a risk of scarring.

“Exfoliation causes too much disruption of the lipid barrier, which holds moisture in the skin, in those who have deficiencies already,” adds Lupo. People with eczema already are prone to dry skin, so if they exfoliate, their skin can crack, creating openings for infection and just generally itching and hurting.

Safer alternatives if you have rosacea or eczema: If your pores are congested or your complexion looks dull, a dermatologist can do a gentle in-office treatment such as manually unclogging pores (a procedure called extraction), microdermabrasion, which aims a spray of tiny crystals onto skin to gently buff away dead skin or photopneumatic therapy, a system that uses suction combined with broadband light to kill bacteria and reduce redness.

Related: Cosmetic Surgery: How Safe Are the Face “Tweaks” You’re Considering?

Be gentle with normal skin too

Even if you don’t have a skin condition like acne, eczema or rosacea, it doesn’t mean you can go to town on your skin with a scrub or brush. Having normal skin is not a license to sandblast it.

“I never recommend scrubs,” says Baxt. Scouring skin can lead to inflammation, impair healing, damage collagen and deplete moisture skin needs to stay healthy.

For gentle but effective exfoliation, use a salicylic acid wash followed by a glycolic acid cream, lotion or peel pad. If you’re wedded to the idea of a scrub because you like the way it feels, choose one with smooth or gentle beads such as jojoba, Baxt advises.

A sonic brush, which uses sonic vibrations to jar dead cells and dirt from skin, is a great exfoliating tool for normal skin. It’s also a good option for sensitive skin — skin that tends to get red or dry easily, stings after you apply certain products or flakes even if it’s oily. If you have sensitive skin, use only the gentle brush head, Lupo advises, and limit sonic brushing to once a day or every other day.

Whatever type of skin you have, always apply a moisturizer or serum after exfoliating.

Related: 6 Smart Ways to Use Coconut Oil On Your Skin