Republican U.S. Senate candidate Sen. Scott P. Brown’s campaign today turned down an invite to debate challenger Jack E. Robinson at the John F. Kennedy Library on Nov. 16, saying they object to the sponsors and the venue.

“They might as well invite us to the Democratic Party headquarters and let (Gov.) Deval Patrick ask the questions,” said Brown communications director Eric Fehrnstrom. “We want a debate with a neutral setting with an impartial and unprejudiced sponsor.”

Republican Jack E. Robinson has accepted the invitation, however, calling it, “a wonderful opportunity for voters to hear the Republican side of the issues,” in a statement.

Robinson’s campaign spokeswoman Alleigh Marre questioned the snub. “We hope he reconsiders. If he can’t face Republicans in November, how is he going to do it in December,” asked Marre.

The debate was supposed to be sponsored by the Kennedy Institute for the United State’s Senate. Peter Meade, the institute’s president, has donated to Attorney General Martha Coakley and U.S. Rep. Michael E. Capuano, two Democratic candidates in the U.S. Senate race.

Fehrnstrom said Brown has agreed to a Nov. 8 debate at the University of Massachusetts at Amherst sponsored by the League of Women voters. The debate will include Democratic candidates. He is also expecting to attend a Boston Chamber of Commerce debate that has yet to be scheduled.

The JFK Library debate was expected to be a high-profile face off between the Republicans with extensive television and radio coverage.

Meade did not immediately return a call for comment.