It’s an insider’s secret that’s about to be revealed. John Smedley’s merino wool polo shirts, with their generous yet perfectly proportioned collar, fine knit and three-button fastening, are flattering to the eye, slip under a jacket or over jeans and, as such, have been a favorite of the elegant for decades. But what’s their secret? To find out, head for Matlock, in Derbyshire, a two-hour train journey from London. When you arrive, you’re in for a complete change of scenery: a small cottage serves as the station, sheep graze in fields of green grass as far as the eye can see, and a little drizzle so British sprinkles us.

Seven kilometers away, the red-brick buildings are reminiscent of England. Lea Mill, founded in 1784 by Peter Nightingale and John Smedley (first of the name), is one of the world’s oldest textile mills, which in its early days spun cotton. In 1819, John Smedley (second of the name) was 16 years old when he joined the company…