The following is from a National Geographic article on the James ossuary:
http://news.nationalgeographic.com/news/2002/10/1021_021021_christianrelicbox_2.html
QUOTE:
Jesus and James

Whether Jesus was the son of God is a theological problem, said Lemaire. But historians don't doubt the existence of either James or Jesus; both are mentioned frequently in early historical accounts.

Following the death of Jesus in 29 A.D., James assumed leadership of the Christian church in Jerusalem until he himself was martyred in 62 A.D. According to biblical accounts, he was one of the first apostles to see Jesus after his resurrection.

He is referred to as the brother of Jesus in both the Bible and in contemporary historical accounts. In Matthew 13:55-56, for instance, Jesus is said to have four brothers and two sisters. But the exact nature of these relationships—whether they were full siblings by blood, half siblings, or cousins—has been open to interpretation.

"If you're Catholic, you think they're cousins because the perpetual virginity of Mary is official church doctrine," said Witheringon. "But there are a lot of problems in the historical record with that."

"When James is referred to as the 'brother of our lord' in the New Testament, the word used means 'blood brother,'" he continued. "It would have to be qualified in context to mean something different."

A second interpretation is that James and the other siblings are half-brothers and -sisters, Joseph's children from a prior marriage.
END QUOTE

This seems to be one of those points that you could argue circles in, depending on what doctrine you have been taught and follow and how you choose to interpret it.