Fetching details from the dictionary...
against the grain (of paper)
(putting) against the grain
「逆目」 has another reading with different meanings.
going against the grain (woodworking); cross-grain