Unless there's some ferry service that doesn't appear in game... where are they coming from? There's a full platform of people in nice clothes that appear. They surely don't work there as they're not in work clothes. The harbor seems to be a very "work" oriented place focused on cargo so I'm not sure why there would be upper class passenger service for unless they are waiting for a ship. Is there anything in the lore about that?
There are 2 local passenger services from Percy's timetable that goes from Ffarquhar all the way to the harbour station and back. (does indeed be weird without the ships at the harbor, ain't wrong there) As for the station itself, it is on the Sodor map, so it might indeed have a use for passengers at the harbor before Tidmouth & Bendam being major harbor spots for the books & the show, respectively.