That hub should wiggle and the outer bearing should pop out, regardless of any brake wear. With that being said, at this point I would make sure those shoes are in all the way just in case. While it doesn't look anywhere near bad enough in the picture, it almost sounds like the inner race spun and welded itself to the spindle. Either that, or its something unknowably odd, like the last person in there noticed a worn spindle and staked, welded, or compounded the bearing in to compensate. That would be incredibly unlikely however...
You might try using a brass punch (and maybe escalating to a chisel, sledge or air hammer) to work on the inner race surface and break free whatever is holding it to the spindle. It could just be a burr or ridge in the spindle.