Jump to content


Referrer Information.

  • Please log in to reply
No replies to this topic

#1 Name Game

Name Game

    Global Board Dad

  • Member
  • 2,421 posts

Posted 16 February 2003 - 01:25 PM

What is all the fuss?

This thread will attempt to give you information utilizing information you can find on the Internet about the issue. If your run a legit game server and want to block certain sites from linking to you it is nice to have but you will end up bocking out many customers.

If your do not want the next site you visit to know where you just came from then referrer is you enemy.


"Cookies" and Privacy
Information about privacy on the Internet: Cookies, Referrer Fields, adware, etc.



Cookies are encoded text files that are stored on your computer by websites that you visit. You are usually not informed of this happening.

You can choose (in your browser settings) whether to accept cookies coming in. Neither major browser will inform you nor give you a choice about cookies being called back to the server. If you have cookies they can be called back.

How cookies function :
Cookies are an easy, low-tech means for websites to keep a record of visitors. When you arrive at a website that uses cookies, the cookie is created or "set". With Internet Explorer it will be an individual file in the Windows\Cookies folder. With Netscape Navigator the cookie will be a single line in the file "cookies.txt", in the Netscape folder. The cookie generally records the current time and a code number.

When you go to another page at the website, the cookie can be called back. By referencing the code number, the website can keep track of what happened on the last page....whether you ordered a product, etc. If a cookie is used for this purpose it will usually "expire" after you leave the website.

Persistent cookies :
In some cases cookies will be "persistent", meaning that they stay on your computer.... They become a way to identify you on your next visit to a website. Persistent cookies can be used to make a website password possible, for example, or to allow customization of the webpage that you see. Using the cookie's code number, a website can reference it's record of your prior visits.

Third party cookies :
Cookies can only be read by the domain from whence they came. In the past this aspect has provided a measure of privacy. For instance, a Yahoo cookie is only read by a website in the Yahoo.com domain. With new advances in commercial exploitation of the Internet, however, this limitation has been circumvented. Through the use of third-party cookies it's possible for one cookie to be read from many websites. The way it works:

An advertising company, such as Doubleclick, maintains the ad banners on a large number of websites. When your browser loads any webpage with one of these ads, it allows the ad company to "set" a cookie on your computer. ( Even though you're not browsing in their domain, the ad is being loaded from their domain. )
The result of this is that the ad company can collect information about your browsing habits on any of the numerous websites that host their ads. Third-party cookies allow one website to have access to all the information you've shared across many websites.

Cookies and Universal User IDs :
A more recent development is the user ID in cookies. See the next section, REFERRER FIELDS AND USER IDs, for details about that.

What does all of this mean? :
Some sites offer a free service in exchange for marketing information. For example, the New York Times website requires that you accept cookies in exchange for freely reading their articles. You simply have to choose whether you want to pay that price.

To be realistic, if you have any persistent cookies you should not count on privacy. With current technology, keeping even one cookie might allow any website to obtain an exhaustive dossier on you ( see next topic ).

Referrer Fields And User IDs

The "referrer field" is information sent by the browser when you arrive at a website. Any website you visit normally receives your IP address, the name and version of your browser, the page you're loading and referrer information. The referrer information tells where you just came from. In general this is benign. It provides a way for website operators to get information such as the URLs of other websites linking to them and how many people are arriving from those links.

With the growing awareness of cookies, more people are blocking or deleting cookies from third party sites, making it more difficult to track online activity between sites. To get around that the referrer field is being exploited to track people.

This issue is rather complex. If you want a full explanation:


The gist of it:

? Microsoft has created a special website, known as "Passport", that assigns a user ID
and records it in a cookie.
[This is the ultimate third-party cookie. All users of Hotmail and MSN have
such a cookie, which contains a user ID number that identifies them to any
MS-owned site. Microsoft makes no secret of this intrusion.....They advertise it
as a service! ]

? Any Microsoft-owned website will set such a cookie.

? Other websites can invisibly redirect your browser, for a moment, to the Microsoft website,
where the cookie can be read and your user ID can be returned in the referrer field.
If there is no such cookie, a new ID number will be returned.

? Such a user ID can be used to link information about you from many sources.

? Thus, having even one cookie might expose a vast amount of personal information
to any website you visit.

? Even without having any cookies, this technique could allow you to be tracked between
websites during a browser session, using the new ID returned in the referrer field.

? There's no reason to think that other companies won't set up similar ID websites.
Selling the data collected could be potentially quite lucrative.

Blocking transmission of referrer fields:

Referrer information can be blocked by using a firewall program.
Regarding specific browsers:

? Internet Explorer: IE does not seem to have a means to block referrer fields.

? Netscape Navigator: Navigator 4 has an obscure setting that will stop referrer fields
from being transmitted. You can download this VBScript that will set it for you.

? Opera: Opera has this setting. The program costs about $40.
The only drawback is that Opera sometimes displays webpages poorly; ignoring line breaks,
leaving color out of table borders, choking on graphics....

Other Privacy Issues

? The ubiquitous web bug
Many commercial sites now have their banner ads supplied by other servers. Sites such as Doubleclick, Akamai, avenueA, ClickXchange.....the list is extensive. If, for example, you visit 5 websites that get their ads from Doubleclick, then Doubleclick has a record of your online activity. This can be cross-referenced to get your identity. If you have a cable modem (or in some cases, DSL) your IP address never changes, making it even easier to track you.

You may be able to see where the elements of a page are loading from by watching the text along the bottom "status bar" of your browser. If you're concerned about this kind of web bug: Some firewalls will let you block connection to particular domains. Some browsers, such as K-Meleon and Mozilla, also have a choice in the preference settings to only load images that are coming from the same domain as the webpage that's loading.

? Download accelerators, etc.
Most programs and services that promise faster browsing or downloads are provided for the purpose of online tracking. Some are spyware. Others require that you allow all online activity to be tracked. Some people may not be bothered by that. If you are, and you use these kinds of programs/services, you should read the privacy agreement and license. (Be aware, too, that most privacy agreements contain disclaimers saying the agreement can be changed at any time without notification. Such a privacy agreement is meaningless.) There is an interesting article on CNET if you want to know more.

? If you use free email or free web access you've likely compromised your privacy.

? In some cases, clicking a link in email to go to a website will send your email address to that site. A solution to that is to copy and paste the URL into your browser.

? Some ISPs have begun selling tracking information to marketers. Check your ISP's privacy statement if you're concerned about that.

? If you're using the AOL browser:
Your browser is actually Internet Explorer but the settings relating to privacy and security may have been hidden. To see the differences open the Windows version of Internet Explorer from the Start menu and find 'Internet Options' in the toolbar dropdowns. Compare that to your AOL "www" preferences settings. You'll probably see that 2 of the settings tabs are missing from the AOL browser settings window. In effect, you've been locked out of your browser settings relating to security, cookies, etc.!

? If you enable Javascript and ActiveX controls in your browser you allow for the possibility of websites browsing through your computer; if your browser settings are connected to your email, such as Internet Explorer and Outlook Express, you'll also leave yourself open to HTML email viruses.

See the "Online Security Tips" page for further information about these issues.

The 'Free' Dilemma

Another situation that's arisen: a trend toward free software that hosts advertising. There are now several hundred ad-supported programs that install special software on your computer, allowing for clandestine contact with an advertising company through your internet connection. This contact is said to be for the purpose of periodically replacing the ads with new ones.
There's been a lot of talk about whether new ads are ALL that's being communicated, but that seems to be missing the point....An advertsing company has installed hidden 2-way communication between YOUR private computer and THEIR office. A visit to Webster's dictionary yields this tidbit:

" wiretap - to tap (a telephone wire, etc.) to get information secretly or underhandedly."

A wiretap is a wiretap! How it's currently being used is hardly the relevant issue.

On the other hand, the programs referred to here are free for the taking. What is that all about......
In many cases, clearly, the free giveaway is a mutual con-game. The marketer holds out a free trinket, hoping to lure a passerby. The marketer's plan is to pickpocket while the passerby is busy grabbing the trinket.The strategy of the passerby is to grab the trinket while keeping his wallet intact. ....So who's cheating who?!

Interestingly, the word 'free' has become ubiquitous in newspapers, in magazines, on TV, on the web and in stores. It would seem that we all hope to get something for nothing, rather than strike a fair deal.

Apropos of that, there's an old saying that you can't cheat an honest man....

Get info. about adware and a download to clean it out:
OptOut - GRC.com

To get an ad company's side of the story:

To read more from an anti-ad site:
Beating Adware

To find out more about privacy in respect to Internet technology:
Electronic Privacy Information Center

For information about firewall software see the "Home Handyman" page.
( If you didn't "come in the front door" and don't see the menu on the left
click here to go to the Joe'Software Homepage. )

But I want to keep certain people from my site who are troublemakers.

"we try to steer people in the right direction, but generally try to avoid re-writing the same tutorials and "help-fix-my-site" posts repeatedly. Please see the links in message number 2 above (in this thread), and this Introduction to mod_rewrite for some very useful information.

To answer your specific question, yes, you can use .htaccess to "sort of" fix your problem. The reason I say "sort of" is that using HTTP_REFERER is not a totally reliable method for controlling access; Many browsers, proxy servers, and packages like Norton Internet Security will block referrer information. So, the best you can do is to allow access for your members, visitors from the other site you mention, and visitors whose browsers provide no referrer information.

If you block referrer-less accesses, you will have a lot of problems with blocking legitimate users, and you'll spend a lot of time playing "help desk." I would suggest allowing referrer-less access, and handling problems with unwanted visitors in another way, such as by blocking their IP addresses.


What can you do about blocking this information being sent?


Browser configuration utility for Mozilla and Netscape
SurfSet is a free utility that provides easy access to several settings in the Mozilla-based browsers: Mozilla and Netscape 7. Some of these settings are not included in the Mozilla Preferences window. Some are hidden in Netscape.

Why Mozilla and why SurfSet?
Odds are that you're reading this in Internet Explorer. Do you know what your security settings are? Do you understand the settings? Do you see popup and popunder ads while browsing? Have you had any trouble with self-installing add-ons to Internet Explorer or had your homepage changed inexplicably? Have you ever tried to block referrer headers in IE?

That's why Mozilla. Mozilla is a very good, Open Source browser that can be downloaded from Mozilla.org. Mozilla is the browser that Netscape is made from. That is, Netscape is Mozilla with an AOL-produced cover on it. Unlike Netscape, Internet Explorer, the AOL browser and the MSN browser, Mozilla is not a commercial browser. It's designed with the user in mind. Mozilla offers easily-configured settings that the other browsers do not allow, such as stopping popup windows when javascript is enabled and blocking images (ads) loading from 3rd-party websites.

Mozilla also has further settings options that can be set up by writing your own configuration file. SurfSet is just a small, simple utility that makes some of those extra settings more accessible. SurfSet does not replace the Preferences settings window in Mozilla. It just provides a few settings options that have not been included in the Mozilla Preferences window and replaces a couple of settings that have been hidden from the Netscape Preferences window.

Since Netscape is actually Mozilla inside, most of the SurfSet functions can also be used with Netscape 7.

Among the settings available: Block referrer headers, change the User Agent setting, turn off auto-complete and "What's Related", disable blinking text, etc.
The download is a zip file containing the SurfSet program and an information file. SurfSet does not need to be installed but it does require the VB6 Runtime. If you use Win95, Win98 1st Edition or NT4 the VB6 Runtime file may not be installed. See the Installation Tips page if you think that you may need to install the Runtime.

Download sset.zip - 28 KB
If I do block all of this referrer stuff could I have problems accessing certain sites?

How to pass referrer information to specific Web pages

You want to allow Norton Internet Security (NIS) or Norton Personal Firewall (NPF) to pass referrer information to a specific Web page.

[b]If you want to allow referrer information to pass to a particular Web page, you must create a rule for it.

NOTE: For instructions on passing referrer information with Norton Internet Security or Norton Personal Firewall 2003, see the document How to pass referrer information to specific Web pages in Norton Internet Security or Norton Personal Firewall 2003.

To create a rule:

1. Open the list of Web sites:
a. Open NIS or NPF.
b. Click Options.
c. Click Internet Security or Personal Firewall. This step is not always needed.
d. Click Advanced Options.
e. Click the Web tab.
2. Add a new Web site:
a. Click Add Site. A new site/domain box appears.
b. Enter the name of the site that you want to receive the referrer information and click OK. That site name appears in the left frame of the Advanced Options window.
3. Configure the new Web site:
a. Click the name of the new site.
b. Click the Privacy tab.
c. Check the "Use these rules for..." box.
d. Change the Referrer from "Block" to "Permit." Your screen should look something like this:

e. Click Apply, and then click OK.
4. Click OK to close the Options window.

How referrer information works
Referrer information is information that a Web site sends to another Web site when you click a link on the first Web site to open the second Web site. When you click a Web page, your browser makes a note of the current page you are on and sends that information to the server of the new Web page. This way, the server for the new Web page knows the last Web page you viewed.

If you block referrer information, the server of the page you are requesting to see does not know what page you saw last. By default, Norton Internet Security (NIS) and Norton Personal Firewall (NPF) block referrer information. However, some Web pages require referrer information before allowing you to view the page.

For example, if you are on www.symantec.com and click www.microsoft.com, the browser sends information to the server www.microsoft.com. The information includes the fact that you are currently viewing the site www.symantec.com. If you block referrer information, the server www.microsoft.com does not "know" that you were previously at www.symantec.com. If the server www.microsoft.com required this information, your browser would not display the Web page at www.microsoft.com, and you might see an error.

Where can this all be tested?

I ran your security check and it says that my browser privacy is 0%. Can you tell me what setting i can change in my browser to increase my privacy?

We consider two key factors in this Browser Privacy test: cookies and referrer. According to your results (0%) your browser fails on both referrer and cookies tests.

Unfortunately neither any firewall nor web-browsers are able to block both cookies and referrer. If you do not care of disabling cookies and blocking distributing the information about sites you have visited (referrer) you should not worry about it. But if you care you should find a personal firewall that would be abe to disable cookies and block referrer.

Edited by TheSentinel, 22 April 2010 - 04:41 PM.