Well, it could be a stray signal from your hardware, though that Apple junk is purported to be fairly good stuff, or it might be a driver issue. Do you even have a joystick driver installed? (I dunno if Debian ships it in kernel or as a module, honestly)
I guess you could try building a custom kernel with all the joystick drivers disabled and see if that helps. Or, you know, switch over to Gentoo and spend all of your time recompiling libjpeg instead of using your computer to download pictures of kittens.