Actually, if half the crew would have been able to swim, that would be a pretty high number. Sailors of that era that could swim were rare.
After all the sea was full of monsters and mermaids - and swimming instructors were really hard to find.
Also, the ships were too big - if you fell overboard, stopping and turning to look for any sailor in the water just wasn't seen as "worth it" back then.