Unfortunately iis default pages won’t work with SharePoint, as they don’t actually exist in the iis folder. SharePoint handles which document to return from the database based on the url – which might be triggering the authentication prompt.

You might want to try this, which looks like it allows you to specify the default welcome page to specific folders:
http://techtrainingnotes.blogspot.ca/2011/06/sharepoint-how-to-change-default-home.html

The other alternative that should work is to use the iis 7 url rewrite module to setup redirects to the default page based on the url of the folder. There’s a bit of information on doing this in my blog post:

http://blogs.visigo.com/chriscoulson/mixed-http-and-https-content-with-sharepoint-2010/

The other option (if it’s an option) is to choose ‘Entire Web Site’ for the anonymous access level – I expect that will fix it as well.

I have one SharePoint site which is enabled for anonymous access.

I have few document libraries and every document library has default.aspx page. If I provide full path in browser, SharePoint doesn’t ask for credentials and it opens anonymously. But if I provide path upto document library, SharePoint ask for credential and then redirect to appropriate default.aspx within document library.

I want to avoid this credential asking pop-up and browser should open appropriate default.aspx page of that document library.

As far as I know, a list can’t inherit the anonymous access settings, at least not through the UI, as they’re not defined at the site level. Perhaps there’s a way through PowerShell, but if there is i’m not familiar with it. The way anonymous access was is set in the permissions does seem pretty wonky – it’s as if it was just bolted on as an afterthought.

http://blog.foreignkid.net/2011/11/iis-7-url-rewrite-http-redirect-to-root-site/