Sometimes the commentary and report is louder than the interview itself. In this case, Elfman is recorded as having strong views on not scoring Spider-Man 3. Here are some reports of this:

Spider-Man composer no more

By Devin Faraci
2005.09.11
Source: http://chud.com/news/4290
Today the junket for The Corpse Bride junket was held in Toronto, at the Toronto Film Festival. The junket in general was short on news - Tim Burton is taking a break, Helena Bonham Carter has a few things in the hopper and Johnny Depp is still knee-deep in Captain Jack Sparrow (his mouth was full of gold teeth). But Danny Elfman dropped a surprise on me when I asked him whether Sam Raimi had told him which villains for whom to consider themes. He's not doing Spider-Man 3, he told me. I asked why.
Elfman: Spider-Man 2 was a miserable experience.
Q: Why? Was it too fast or –
Elfman: It’s a complicated thing.
Q: Why was it complicated?
Elfman: My connection to Sam got completely severed. As far as I’m concerned, he went to sleep and somebody put a pod next to him and when he awoke, he wasn’t the same person I’ve known for a decade.
Q: Will you work with him again?
Elfman: No. He went from right there, number 2 on my list of favorite directors, to the last – to the exact opposite of everything I look for in a film experience. Everything I could do on Spider-Man 1 I couldn’t do on Spider-Man 2. He got so intensely attached to the temp music, I couldn’t even adapt my own music. I couldn’t get close enough to me.
It’s the first time I’ve ever walked from a director in twenty years, and hopefully the last time. He became intolerable. I’ve been on some heavy duty films, so to say that it had to be pretty bad. I have been in war zones you wouldn’t believe in 55 films. But this is the first time I’ve said, I’ve had it. It’s just not worth it. I would rather go back to waiting tables than to do Spider-Man 2 again.

Elfman & Raimi Split On Spider-Man

Source: http://www.darkhorizons.com/news05/050913a.php, again reportedly via About.com
2005.09.13
By Garth Franklin
A surprisingly outspoken Danny Elfman revealed to About.com that he's had a falling out with Spider-Man director Sam Raimi and won't be back to compose the music for the third film.
"I'm not working on Spider-Man 3. I'm out of that. I won't miss not doing it. Spider-Man 2 was a miserable experience. It's like my connection with Sam got completely severed. As far as I'm concerned, he went to sleep, somebody put a pod next to him and when he awoke, he wasn't the same person I'd known for a decade" says Elfman. "He went from right there number two on my list of favorite directors to the exact opposite of what I look for in a film experience which is everything I could do on Spider-Man 1, I couldn't do on Spider-Man 2. He got so intensely attached to the temp music that I couldn't even adapt my own music close enough. Let me put it that way. I couldn't get close enough to me, least of all anybody else who was in the temp score."
You will hear Elfman's theme in Spider-Man 3 in any case because Raimi owns it - "He can do whatever he wants. It's the first time I've ever walked from a director in 20 years and ho[p]efully the last time I have to turn my back on somebody but it became like intolerable. And I've been on some heavy duty films, so to say that, it had to be pretty bad. I've been in war zones you couldn't believe in 55 films but this is the first time I said, 'I've had it. It's just not worth it. I'd rather go back to waiting tables than to do Spider-Man 2 again, to have to have the same experience."

Fred Topel of http://actionadventure.about.com/od/spiderman3/f/aaspidey3faq5.htm reported similar news from the About.com source, complete with identical typo in the hurriedly pasted text.
Back to The Elfman Zone