To a large extent the browser decides what to do when a user clicks a PDF link, not the website. For example, Chrome forces you to download the file and open it in the desktop PDF viewer (like Adobe Reader), but other browsers open it within the browser window if a PDF viewing plug-in is installed.
You can, however, tell the browser to open the PDF file in a new window/tab by editing the HTML template. Open the database table gs_BrowserTemplate and look for the record where MimeType='application/pdf'. Then edit the hyperlink in the HtmlTemplate column to include target="_blank", like this:
<p><a href="{MediaObjectUrl}" target="_blank">Enlarge PDF to fit browser window</a></p><iframe src="{MediaObjectUrl}" frameborder="0" style="width:680px;height:600px;border:1px solid #000;"></iframe>
Restart the IIS app pool after editing the record to force the app to reload the template. You can make other changes to further customize the rendering as desired.
Roger Martin
Creator and Lead Developer of Gallery Server Pro