Fetching details from the dictionary...
going against the grain (woodworking); cross-grain
「逆目」 has another reading with different meanings.
against the grain (of paper)