The Minister's Black Veil

Nathanial Hawthorne-The Minister's Black Veil
Page-53 in reader

"Such was the effect of this simple peice of crape that more than one woman of delicate nerves was forced to leave the meeting-house. Yet perhaps the pale-faced congregation was almost as fearfull a sight to the minister, as his black veil to them".

Hawthorne is saying that the people were scared by such a simple thing, and acted as if it had suddenly turned a good man bad. They didnt consider that their reation to it may have been even more scary for the minister. It is foolish to assume, as they did, that there was no reason for the veil, and immediately assume that he had gone mad.

Nathanial Hawthorne-The Minister's Black Veil
Page-55 in reader

"How Strange," said the lady,"that a simple black veil, such as any
woman might wear on her bonnet, should be such a terrible thing on Mr. Hooper's face."

This lady does not see the veil itself as the scary thing here. She instead is scared because he is such a holy man, and he is wearing the attire of a mourning woman. Many people are also upset because it separates them from Mr. Hooper in a sinister way.