With the portable devices widely used, the security problems such as information leakage caused by them have been paid more and more attention. According to the deficiency of the current security management for portable devices based on username and password, this paper presents a security management scheme based on the smart card technology. In this scheme, the fingerprints are used to authenticate the identity of a device’s user, at the same time, the mutual authentication between portable devices and the accessed terminals can be achieved, which prevents the security incidents caused by portable devices at the root.