Regardless of if the addition of the phrase was violating the First Amendment at the time is irrelevant. What's done is done. Yes, the phrase can be removed, but that would also be a violation of the First Amendment. There isn't a tradeoff when it comes to the law. Just because the government didn't see the addition as a violation of the First Amendment back in 1954 doesn't give the present-day government the right to violate it again by taking out the phrase.
In 1943, the decision in Minersville School District v. Gobitis was overturned. In West Virginia State Board of Education v. Barnette, the US Supreme Court ruled that the right to freedom of religion and speech do not apply. Felix Frankfurter wrote that "[t]he mere possession of religious convictions which contradict the relevant concerns of a political society does not relieve the citizen from the discharge of political responsibilities." Meaning that having a cohesive nation is more important than one person's political beliefs.