the "swarthy" term refers to his tan.
but in this quote, it says that he's still paler than the others, despite that he's tanned. meaning that he hasn't tanned as much as the other boys, meaning that his original skin is paler than usual.
and this quote is not quite at the beginning of the book.
i agree that he “was not noticeably darker” than when he originally landed on the island. like you said, it means that his tan is most likely less obvious than the others; however, it doesn’t prove that he was “paler than them.” it just means that his skin wasn’t noticeably darker than it was originally.
furthermore, roger’s face is technically described as “gloomy” not “pale.” the word gloomy is defined as such, according to this source:
before you say anything, i know gloomy doesn’t equal to having dark skin. in context, gloomy was used to reflect roger’s dismal mood, his emotional state rather than his physical one.
however, you can’t use gloomy as a synonym for being pale/light skinned.