The operator of the Nasdaq market withdrew its $11 billion rival bid for the New York Stock Exchange’s parent Monday after the Justice Department said it would challenge the deal, raising questions about Nasdaq’s next steps as exchanges around the world merge to remain competitive.

Nasdaq, fierce rivals with the NYSE for years, had sought to derail a planned merger between NYSE Euronext and the Deutsche Boerse. In April, Nasdaq OMX joined forces with the IntercontinentalExchange, known as ICE, a commodities futures and derivatives marketplace, to make a bid for NYSE Euronext. Under their proposal, Nasdaq would take the Big Board, while the IntercontinentalExchange would get the NYSE’s derivatives business.