Touch in the Wild: Learning Fine-Grained Manipulation with a Portable Visuo-Tactile Gripper